💡 ❗ ⭐ 😎 ❤ 😀 hi all, this post spent a long time in the works/ “drawing board” building up & now finally decided to “clean house” after it all finally seemed to reach “critical mass” at least in my bookmark pile. it is difficult to time it. empirical CS/ math has been a big topic in the past on this blog.**[a15][a16]** this can be a controversial area for multiple reasons/ “standard objections”. there are occasional large breakthrus in this area. however, they are not widely known/ heralded and sometimes even quickly forgotten. one might imagine the momentum steadily building in this area but it seems to be very uneven in progress/ attention. over the last few years here are a few key experimental breakthrus reported but not widely appreciated:

- decrease of the gap in the Zhang twin primes proof. some of this work was done with experimental/ computer searches that found smaller gaps which were then verified/ proven mathematically. its as if numerical evidence gives theoreticians confidence to find/ nail down abstract proofs (instead of hunting for something that might not exist).
- the big-award-winning Kadison Singer proof by Spielman et al.; was aware of this proof but was not even aware myself of the empirical component until the recent writeup by Klarreich for the Simons Quanta magazine.
**[b]**after this account, one wonders if the proof would even have been found/ possible without computer experiments! she writes: (this reminds me of binet’s formula for fibonacci numbers!)

Spielman, Marcus and Srivastava suspected that the answer was yes, and their intuition did not just stem from their previous work on network sparsification. They also ran millions of simulations without finding any counterexamples. “A lot of our stuff was led by experimentation,” Marcus said. “Twenty years ago, the three of us sitting in the same room would not have solved this problem.”

The simulations convinced them that they were on the right track, even as the problem raised one stumbling block after another. […]

A network’s electrical properties are governed by a special equation called the network’s “characteristic polynomial.” As the trio performed computer experiments on these polynomials, they found that the equations seemed to have hidden structure: Their solutions were always real numbers (as opposed to complex numbers), and, surprisingly, adding these polynomials together always seemed to result in a new polynomial with that same property. “These polynomials were doing more than we gave them credit for,” Marcus said. “We used them as a way of transferring knowledge, but really the polynomials seemed to be containing knowledge themselves.”

- the Erdos Discrepancy Problem results by Konev/ Lisitsa. this got a lot of attention and plausibly played a role in the recent proof by Tao. again, the pattern of solid numerical results found by experimentalists give mathematician(s) confidence to tackle previously un-conjectured results.
- a new pentagonal tiling has been found and caused major surprise/ ripples in the area.
**[r]**

“We discovered the tile using using a computer to exhaustively search through a large but finite set of possibilities,” said Casey. “We were of course very excited and a bit surprised to find the new type of pentagon.

- Fermi-Pasta-Ulam nonlinear physics/ applied math problem recently subject to major advance.
**[q4]**

“Thanks to this lateral approach, we partially answered the 60-year-old FPU problem. We were able to predict the long thermalization timescale knowing the initial conditions of the system. We also corroborated our theoretical result with extensive numerical simulations.

😮 the excellent/ striking Kadison-Singer writeup by Klarreich and the nearly-eyepopping excerpt above was one trigger for this post. how many people were aware of their central/ pivotal computer experiments? are they even referred to in their writeups anywhere? it is likely buried somewhere if mentioned at all, and that is something of a shame. but, this is not surprising/ typical and par for the course in this area.

the new detailed behind-the-scenes account also makes one wonder if mathematicians or “CSists” were a little more flexible and experimental whether major new insights would be possible. after many years of studying this, think most lack of progress in hard problems is mainly due to their inherent difficulty, but maybe at least some small part is psychological, sociological, cultural, and conventional.

mathematicians and computer scientists routinely express ignorance and at times even some disdain for coding/ experimentation. the title of that CS stackexchange question is *“how can I teach CS without computers?”* wow, a near zen question if I ever heard one! sounds to me sort of like, “what is the sound of one hand clapping?” or, “how do I teach bicycle riding without using bicycles”? this all seems near/ reminiscent of the semifamous/ notorious (to me) quote attributed to Dijkstra, “CS is no more about computers than astronomy is about telescopes.” my contrarian position to that would be simply that *“advanced astronomy is impossible without telescopes”* and *“the better the telescopes, the better the astronomy.”*

astronomy did advance during and prior to the dark ages, but it was (a lot of tautologies around here!) *dark and pre-dark age astronomy, also pre-enlightenment “science”!* however, all that said, do very much like the *yin-yang* theoretical/ applied dichotomy/ juxtaposition evident in both fields though, and the analogy/ parallel of exploration of *terra incognita* in synergy between mind and matter, knowledge and tools, and theory/ application.

so have been waiting for awhile for a big empirical breakthru in CS/ math that makes the front pages of science journals. but this has not materialized. on the other hand my links runneth over. so am going to get out ahead of the pack on this one and write all this up, and also in some anticipation of major/ imminent advances in the area, but (alas) nothing in particular at the moment.

my list of favorite/ notable/ leading *(“hall of fame”)* CS empirical researchers in case anyone questions that angle:

- founder of the field Knuth did a lot of empirical work in the Art of Computer Programming. some of this has been lost in memory. wish someone (or he himself!) would do a survey of his empirical work.
- Doron Zeilberger is famous for his empirical work.
**[h6]** - Sedgewick has spoken/ written eloquently on the role of empirical aspects of algorithmic research.
**[a4][a5]** - Lipton has also blogged on this “flip side” somewhat under the topic of “galactic algorithms” ie unwieldy algorithms that are not implemented in practice, whether program size matters, etc.
**[n]** - CS “forefather” Turing himself had a very applied side, eg in inventing/ working on one of the worlds 1st computers during WWII. this story was not very well known previously but very deservedly got the 7-oscar-nominated treatment in Imitation Game.
- “imho” one of the greatest ever empirical theoreticians (yes, its an oxymoron) is the living Wolfram with his ambitious, meticulous, copious, monolithic, near-encyclopedic inquiry into cellular automata. alas, hes a highly controversial and polarizing figure at times, something like the “rodney dangerfield” of researchers with many other academics and scientists criticizing his style etc.

some neat hilights, Mitzenmacher has recently emerged as a big proponent/ advocate/ defender with a big essay in the CACM.**[a6][m1]** the Simons institute and Klarreich are hot on the tail of empirical approaches in crosscutting areas of CS/ math/ applications.**[a9][b11][h2][h6] **re CACM profile/ advocation see also** [h3]**.

the concept of “galactic algorithms” seems a bit slippery. it originally seemed to be coined to refer to very complex algorithms that have good theoretical performance. but then there is the concept of implementation difficulty that inevitably creeps in. so “galactic algorithms” term (only relatively recently coined by Lipton) seems to me to be subject to the “elephant and the blind men” aka “rashomon” phenomenon.

it is a bit of a philosophical distinction in some ways whether a theoretical algorithm is implemented, and in other ways its a very practical, pragmatic question. this is the CS version of “long proofs” eg from mathematics like the Classification of Finite Simple Groups which spans thousands of research pages and which (mostly/ apparently) has never actually been “implemented” in code.**[m4]**

so ideas about program verification and proof correctness/ verification also enter here.**[h]** it is somewhat taboo to talk about erroneous proofs/ papers in either CS or math, but from an implementers pov, this is a key aspect of *human!* science.

there is a lot of innovation in empirical approaches, eg notably in sorting networks charging ahead,**[e]** and eg in an area dear to my heart, SAT, namely ASP and SMT programming.**[i][j]** (oh, and they are related; the new sorting network results/ proofs are largely based on SAT solvers.)

have very long thought/ proposed (~2 decades!) that studying SAT instances empirically could lead to major insight into problem hardness esp with factoring, but this work is mostly not even undergone by the theoretical/ applied community because its a mostly crosscutting idea that tends to “fall through the cracks”.**[g]** however **[c14]** is a 100p(!) masters thesis by Herwig that does innovative/ creative/ groundbreaking work in the area, which seems not to have been cited or advanced on much.

there is a US govt recognition/ funding initiative called NSF “AitF/8F”, “algorithms in the Field,” so now with *real dollars* involved you *know* its real.**[k]**

memcomputing has some wild claims and most theoretical researchers are not taking it too seriously (Aaronson has lambasted it), but, personally avoiding knee-jerk reaction, it looks to have innovative elements and promise to me and think there is some irrational pushback/ overreaction typical of the “anti experimental bias” that pervades “classic” math/ CS.**[l]** my thinking is that maybe there are tricky/ “slow” scalability or noise issues on the efficiency claims that are difficult to untangle/ observe at “low datapoints”. that may sound like handwaving but its *nearly exactly* the same case for quantum computing—exactly the kind of thing that interdisciplinary, theorist/ applied-crosscutting-mix experts can help with.

overall this whole picture combined represents a paradigm shift in math and computer science, but few have noticed or documented it overall.**[a][h]** it appears to me despite decades of use/ application in the area, the use of computers in science research is probably in its infancy and its penetration/ increase is part of a very long wave spanning many decades, easily more than a ½ century.

it may seem counterintuitive but in my thinking, but maybe not within the reductionistic/ overspecialized/ compartmentalized tendency of modern science, both Big Data and Deep Learning are unequivocally aspects of empirical computer science. this gives the overall field some major/ intense rocket fuel/ full engine/ throttle/ thrust ahead! deep learning is being used mainly for object and speech recognition advances, but think it may lead to mathematics and CS research advances in the not-so-distant future. this is not completely outlandish and has been long envisioned by other visionary authors/ authorities, eg most notably Gowers (see earlier blog on that).**[a16]**

addendum: **[p]** are refs turned up with google. several (about ½ dozen) classes/ syllabuses around the internet. speaking of “long forgotten,” **[p1]** is a Turing Award lecture by Newell and Simon from 1975, “Computer Science as Empirical Science”. their award was for “basic contributions to artificial intelligence, the psychology of human cognition, and list processing” but they took the opportunity to emphasize the empirical aspects of computer science.

addendum2: **[q][r]** connections to mathematics such as the recent periodic pentagon tiling proof found empirically, theorem proving and verification, etc.

**a. experimental2****b. spielman/kadison/singer****c. papers/ books****d. books****e. sorting networks****f. stackexchange Q/A****g. fact2sat****h. empirical math****i. asp****j. smt****k. NSF Algorithms in the Field “8F”****l. memcomputing****m. powerful/ complex/ galactic/ practical/ erroneous****n. lipton****o. (SE Q/A) correctness/ implementation****p. experimental3****q. computer math****r. pentagon tiling**

**a. experimental2**

- 1. Shtetl-Optimized » Blog Archive » Do theoretical computer scientists despise practitioners? (Answer: no, that’s crazy)
- 2. Applying theory to practice (and practice to theory) | MIT CSAIL Theory of Computation
- 3. Four Fifths = A Fifth (Eulers conjecture counterexample) | bit-player
- 4. Putting the Science back in Computer Science/ Sedgewick
- 5. Algorithms for the masses / Sedgewick
- 6. Theory Without Experiments: Have We Gone Too Far? | September 2015 | Communications of the ACM
- 7. Our number’s up: Machines will do maths we’ll never understand | New Scientist
- 8. Sanjeev Arora on rethinking the graduate algorithms course | Windows On Theory
- 9. Chaos Theory in Ecology Predicts Future Populations | Quanta Magazine
- 10. Can computer science untangle our transit maps? | MIT CSAIL
- 11. Quasi-empiricism in mathematics – Wikipedia, the free encyclopedia
- 12. The Unreasonable Effectiveness of Mathematics in the Natural Sciences – Wikipedia, the free encyclopedia
- 13. In silico – Wikipedia, the free encyclopedia
- 14. Where Mathematics Comes From – Wikipedia, the free encyclopedia
- 15. great moments in empirical/experimental math/TCS research, breakthrough SAT induction idea | Turing Machine
- 16. adventures and commotions in automated theorem proving | Turing Machine

**b. spielman/kadison/singer**

- 1. Effort to model Facebook yields key to famous math problem (and a prize)
- 2. [math/0510024] The Kadison-Singer Problem in Mathematics and Engineering
- 3. Daniel Spielman — MacArthur Foundation
- 4. YaleNews | Yale computer scientist named MacArthur Fellow
- 5. YaleNews | Yale Computer Scientist Wins Prestigious Nevanlinna Prize
- 6. YaleNews | Computer scientist Daniel Spielman named inaugural Simons Investigator
- 7. YaleNews | Yale’s Spielman Wins Gödel Prize for Showing How Computer Algorithms Solve Problems
- 8. Yale Scientific Magazine – Analyzing Algorithms: Computer Scientist Daniel Spielman Wins MacArthur Genius Grant
- 9. Daniel Spielman Wins MacArthur ‘Genius’ Award | blog@CACM | Communications of the ACM
- 10. Indian researcher helps prove math conjecture from the 1950s
- 11. Computer Scientists Solve Kadison-Singer Problem | Quanta Magazine
- 12. ‘Outsiders’ Crack a 50-Year-Old Math Problem | WIRED

**c. papers**

- 1. [1404.6248] The Nature of Scientific Proof in the Age of Simulations
- 2. [1403.0734] Counting small cliques in MapReduce
- 3. [1403.6602] Pivot Sampling in Java 7’s Dual-Pivot Quicksort
- 4. [1007.2295v1] Phase Plots of Complex Functions: a Journey in Illustration
- 5. [1410.6771] Spectral Experiments+
- 6. [1410.8202] Binary Determinantal Complexity
- 7. [1412.3333] Empirical Algorithmics: draw your own conclusions
- 8. On the boundaries of solvability and unsolvability in tag systems. Theoretical and Experimental Results. / De Mol
- 9. [1508.05737] When Six Gates are Not Enough
- 10. Minimal Elements for the Prime Numbers
- 11. [1309.1779] Fractal dimension versus computational complexity
- 12. [1511.06620] Use of Eigenvector Centrality to Detect Graph Isomorphism
- 13. NEURAL GPUS LEARN ALGORITHMS/ Kaiser, Sutskever
- 14. Decomposing satisfiability problems/ Herwig

**d. books**

- 1. Algorithmic Number Theory / Bach, Shallit
- 2. Algorithm Engineering: Concepts and Practice / Chimani, Klein
- 3. Computational Number Theory / Pomerance
- 4. Prime Numbers – A Computational Perspective
- 5. An Introduction to Sieve Methods and Their Applications (London Mathematical Society Student Texts): Alina Carmen Cojocaru, M. Ram Murty: 9780521612753: Amazon.com: Books
- 6. Algorithm Engineering: Concepts and Practice Chimani/Klein
- 7. A Guide to Experimental Algorithmics: Catherine C. McGeoch: 9780521173018: Amazon.com: Books
- 8. Computer Science in the Information Age – Springer

**e. sorting networks**

- 1. [1405.5754] Twenty-Five Comparators is Optimal when Sorting Nine Inputs (and Twenty-Nine for Ten)
- 2. [1410.2736] Faster Sorting Networks for $17$, $19$ and $20$ Inputs
- 3. [1411.6408] Sorting Networks: the End Game
- 4. [1412.5302] Optimal-Depth Sorting Networks
- 5. [1501.06946] New Bounds on Optimal Sorting Networks
- 6. [1502.04748] Towards Optimal Sorting Networks: The Third Level
- 7. [1502.05983] Sorting Networks: The Final Countdown

**f. stackexchange Q/A**

- 1. soft question – Examples of unexpected mathematical images – MathOverflow
- 2. automata theory – Finding the smallest DFA that separates two words without using brute force search? – Theoretical Computer Science Stack Exchange
- 3. geometry – Is this Batman equation for real? – Mathematics Stack Exchange
- 4. Research before surveying the literature – is it ever a good idea? – Academia Stack Exchange
- 5. soft question – Constraint satisfaction problem (CSP) vs. satisfiability modulo theory (SMT); with a coda on constraint programming – Theoretical Computer Science Stack Exchange
- 6. algorithms – Distinguish Decision Procedure vs SMT solver vs Theorem prover vs Constraint solver – Computer Science Stack Exchange
- 7. education – How can I teach computer science without using computers? – Computer Science Stack Exchange
- 8. Is anyone aware of a counter-example to the Dharwadker-Tevet Graph Isomorphism algorithm? – Theoretical Computer Science Stack Exchange
- 9. ds.algorithms – A reading list on experimental algorithmics – Theoretical Computer Science Stack Exchange
- 10. ds.algorithms – How to evaluate and compare the performance of algorithms in practice? – Theoretical Computer Science Stack Exchange

**g. fact2sat**

- 1. CNF Generator for Factoring Problems
- 2. ToughSAT Generation
- 3. cc.complexity theory – Fast Reduction from RSA to SAT – Theoretical Computer Science Stack Exchange
- 4. complexity theory – Can top SAT-solvers factor easy numbers? – Computer Science Stack Exchange
- 5. reductions – Reducing the integer factorization problem to an NP-Complete problem – Computer Science Stack Exchange
- 6. algorithms – CNF Generator for Factoring Problems – Computer Science Stack Exchange
- 7. boolean algebra – Convert Circuit SAT to 3-SAT – Mathematics Stack Exchange
- 8. algorithms – Converting (math) problems to SAT instances – Computer Science Stack Exchange
- 9. fl.formal languages – Translation of context-free parsing into SAT – Theoretical Computer Science Stack Exchange

**h. empirical math**

- 1. The Little Prover / Friedman, Eastlund
- 2. Will Computers Redefine the Roots of Math? / quanta magazine
- 3. Verifying Computations without Reexecuting Them | February 2015 | Communications of the ACM
- 4. The Revolution Will Not Be Formalized | The n-Category Café
- 5. In Mathematics, Mistakes Aren’t What They Used To Be: Computers are changing the way proofs are done
- 6. As Math Grows More Complex, Will Computers Reign? / Wolchover, Quanta mag, Wired

**i. asp**

- 1. A User’s Guide to gringo, clasp, clingo, and iclingo
- 2. Potassco: The Potsdam Answer Set Solving Collection
- 3. reference request – looking for notable applications of ASP (Answer Set Programming) in TCS – Theoretical Computer Science Stack Exchange
- 4. np hardness – Transitive feedback arc set (TFAS): NP-complete? – Theoretical Computer Science Stack Exchange
- 5. Potassco – the Potsdam Answer Set Solving Collection (Sourceforge)
- 6. Answer set programming – Wikipedia, the free encyclopedia

**j. smt**

- 1. verification – Limits of SMT solvers – Stack Overflow
- 2. reference request – Recipe book for SAT encodings? – Computer Science Stack Exchange
- 3. Satisfiability modulo theories – Wikipedia, the free encyclopedia

**k. NSF Algorithms in the Field “8F”**

- 1. nsf.gov – Funding – Algorithms in the Field – US National Science Foundation (NSF)
- 2. DIMACS 2011 Algorithms in the Field video playlist
- 3. Algorithms In The Field (8F)

**l. memcomputing**

- 1. Meet The Memcomputer: The Brain-Like Alternative to Quantum Computing
- 2. Memcomputing NP-complete problems in polynomial time using polynomial resources and collective states | Science Advances
- 3. Shtetl-Optimized » Blog Archive » Memrefuting
- 4. [1512.05064] Polynomial-time solution of prime factorization and NP-hard problems with digital memcomputing machines

**m. powerful/ complex/ galactic/ practical/ erroneous**

- 1. My Biased Coin: What’s Important in Algoirthms (re Knuth interview)
- 2. List of long proofs – Wikipedia, the free encyclopedia
- 3. Cosmological Lower Bound on the Circuit Complexity of a Small Problem in Logic / Stockmeyer, Meyer
- 4. Researchers Race to Rescue the Enormous Theorem before Its Giant Proof Vanishes – Scientific American

**n. lipton**

- 1. An Old Galactic Result | Gödel’s Lost Letter and P=NP
- 2. Does Program Size Matter? | Gödel’s Lost Letter and P=NP
- 3. A Matchless Result | Gödel’s Lost Letter and P=NP
- 4. Gödel’s Lost Letter and P=NP | a personal view of the theory of computation
- 5. Practically P=NP? | Gödel’s Lost Letter and P=NP
- 6. Galactic Algorithms | Gödel’s Lost Letter and P=NP
- 7. Speedup in Computational Complexity | Are there familiar computational problems with no best algorithm?

**o. (SE Q/A) correctness/ implementation**

- 1. soft question – Are research papers hard to read? – Theoretical Computer Science Stack Exchange
- 2. ds.algorithms – Powerful Algorithms too complex to implement – Theoretical Computer Science Stack Exchange
- 3. ds.algorithms – Powerful Algorithms that are just too Hard to Implement– How to be sure They are Right? – Theoretical Computer Science Stack Exchange
- 4. soft question – Long-lasting errors in computer science – Theoretical Computer Science Stack Exchange
- 5. ho.history overview – Widely accepted mathematical results that were later shown wrong? – MathOverflow
- 6. proofs – Famous computer science results which correctness is uncertain? – Theoretical Computer Science Stack Exchange
- 7. proofs – Famous computer science results which correctness is uncertain? – Theoretical Computer Science Stack Exchange
- 8. ds.algorithms – Powerful Algorithms that are just too Hard to Implement– How to be sure They are Right? – Theoretical Computer Science Stack Exchange
- 9. soft question – Long-lasting errors in computer science – Theoretical Computer Science Stack Exchange
- 10. Difference between time complexity and computational complexity – Computer Science Stack Exchange

**p. experimental3**

- 1. Computer Science as Empirical Inquiry: Symbols and Search / Newell/ Simon (1975 Turing Award Lecture)
- 2. Research Methods for Empirical CS/ Jensen, UMass
- 3. Weaving Empirical Research Into Computer Science Education / Kaur, UNC
- 4. Empirical Methods | Emery Berger
- 5. CSC2130 Empirical Research Methods for Computer Scientists/ Easterbrook, Toronto
- 6. CS 294-105: Empirical Analysis/ Paxson, ICSI
- 7. Empirical Research Methods in Computer Science/ Smith, Johns Hopkins
- 8. Empirical investigation throughout the CS curriculum/ Reed, Miller
- 9. CiteSeerX — Toward Empirically-Based Software Visualization Languages
- 10. What computational physics is really about/ Wired

**q. computer math**

- 1. Applied Mathematics is Bad Mathematics/ Halmos
- 2. Billionaire Mathematician – Numberphile – YouTube
- 3. Workshop on Type Inference and Automated Proving
- 4. Mathematicians solve 60-year old-problem — ScienceDaily

**r. pentagon tiling**

- 1. Attack on the pentagon results in discovery of new mathematical tile | Science | The Guardian
- 2. Scientists discover a new shape by trying to solve one of maths’ most complex problems | Daily Mail Online
- 3. Historic ‘Tile’ Discovery Gives Math World A Big Jolt
- 4. Scientists Discover 15th Convex Pentagon Able To Tile A Plane : NPR