A breakthrough in physics is rarely a better fit to the data nowadays.

The gap

The ladder

Six rungs, ordered. Reinterpretation is L1.

Examples are Dirac Medals, read against what the citation actually says.

L0 Regression

Keep the description you have and solve inside it. Dirac Medal 2025 — Kerr. An exact rotating black-hole solution of Einstein’s equations. Nothing about general relativity changed; he solved a hard problem in the language it already had. The only rung machine learning has an operation for — and not the easy one.

L1 Reinterpretation

The theory predicts the same numbers but carries less. Heineman Prize 1983 — Kruskal. The surface at r = 2M in the Schwarzschild solution was taken for a physical singularity, a place where spacetime breaks down. Kruskal’s coordinates showed there is nothing there: it is an artifact of the chart, not of the spacetime. Identical predictions, and a supposed feature of the world deleted — the same shape as Einstein dropping the aether. 38 events — the rarest move in the corpus.

L2 Retyping

Change the level, or the quantity, the description is written in. Dirac Medal 2024 — Casini, Huerta, Ryu & Takayanagi. Stop measuring raw entanglement entropy, which diverges, and measure relative entropy instead. Same physics, a different quantity counted as the thing. Most description-changing work is here — and it is the least coherent rung. Of its 501 events, 34% change the level and only 7% merge or split categories, which is how we first defined it. It should probably be two rungs.

L3 Transfer

Carry one field’s language into another. Dirac Medal 2019 — Mukhanov, Starobinsky & Sunyaev. The citation says it outright — “combining microscopic physics with the large scale structure of the universe.”

L4 New object

Add a kind of thing the language did not have. Dirac Medal 2020 — Neveu, Ramond & Virasoro. Cited for work “which introduced new Bosonic and Fermionic symmetries into physics.” The prize text names the move.

L5 New move

Change what counts as a move at all. Cohen 1966 — forcing. Not a physics award; none reaches this rung. A general method for building models to order — a new kind of move rather than a new result. One candidate in 1,547 events, and the annotator flagged even that one uncertain.

What automated discovery does today

Fourteen systems. Everything piles up on the bottom rung.

Each system is placed on the ladder above by the operation it exposes, not by the significance of its results. The claim is about published operations, never about limits in principle.

Nothing performs L1. No system takes a theory that already fits and returns a theory that fits identically while carrying less — the Lorentz-to-Einstein move. And a language model does not close it: on five episodes it scored 87% when it could recognise them and 48%, chance, when the names were stripped out. It recalls; it does not judge.

So we tried to build one

If nothing performs the move, can it even be written down as something a machine runs?

That is the question the rest of this page is about. A move you can only recognise in hindsight, in prose, is not something you can automate. So the first job is to turn “Einstein’s theory carried less than Lorentz’s” into a quantity you can compute from a theory and its data, before anyone knows who was right.

1 A theory is a set of knobs

Write any theory as something with adjustable parts — masses, couplings, whatever it takes to make a prediction. Now turn each knob a little and watch every prediction it makes.

2 Some combinations of knobs move nothing

Sometimes you can turn a whole combination of knobs and not one prediction changes. That combination is carrying no information about the world. It is doing no work. It is an aether. Lorentz’s aether velocity was exactly this: a number in the theory that could take any value without altering a single measurable quantity.

3 Reinterpretation is deleting it

Einstein’s move, stated mechanically: find the combinations that move nothing, and remove them. Predictions stay identical — that is what makes it a reinterpretation and not just a better fit. Formally these are the null directions of the prediction Jacobian; the machinery is standard identifiability analysis, pointed at theory choice instead of at a fit.

4 Measured against the error bars, not exactly

Lorentz’s aether was not undetectable in principle — it was undetectable to the precision of the instruments of the day. So the real question is whether a knob can be turned by a large amount without shifting any prediction further than its own error bar.

One extra question separates them, and it is about locality: can you apply the change at each point independently, or only to the whole universe at once?

One test, two discovery moves. The second is the more productive of the pair: a distinction no instrument can see, but only when you move it everywhere at once, is a conservation law. Our implementation now refuses to delete anything at all when it has no way to run that locality test, rather than guess.

One piece is still missing. This machinery can only ever reject candidates — something has to propose them, and it can only find structure inside the vocabulary it was handed. That is the job we give the language model, and only that job: it generates, it never decides. Its own scorecard is the reason — 87% when it recognised the episode, 48% when the names were stripped.

Three more, built the same way

The same shape — one obvious test, plus a second that makes the answer pay — turns out to fit three further rungs.

L4 consumes what L1 produces: the symmetry branch finds conserved charges, and L4 uses one — when a found charge fails to balance, the missing amount describes something not yet named. That is Pauli's move, and it recovers a hidden quantity to under 1%.

What building four of them showed

The obvious test accepts empty answers. The refutable one is what does the work.

Each operation was designed with a natural acceptance test. In three of the four that test turned out to accept answers that are empty; the fourth fails in the opposite direction. We did not design this pattern in — it showed up three times independently.

The clearest case is coarse-graining. A reduction that throws away everything predicts its own future perfectly — there is nothing left to predict. So does one that keeps only the fastest-decaying parts, which all decay to zero: a description of nothing, evolving correctly. Both satisfy the first test. Only the second refuses them.

Corrected 31 Aug 2026. An earlier version of this page reported the 5-of-6 figure without saying it was measured in sample. A reviewer asked; re-scoring on held-out states gives 2 of 6. The finding stands in weaker form and the shipped code now scores it held out.

The same lesson turned on us. Our own second guard was wrong three times — built on features that could not represent the measurements, then overfitted to a perfect in-sample score on 62 samples, then swamped by scale differences. Every one was caught by an invariant, never by a number that looked wrong. The version we kept must score exactly 1.000 when the reduction keeps everything. It does.

Does it work?

Three questions, in order: does the code do what it claims, does it work on real physics, and where does it fail?

Taking them out of order is how you fool yourself. So: first controls that cannot pass by accident, then a real physical system, then a live scientific dispute where nobody knows the answer.

First — does the code do what it says?

Twenty-one checks on cases where we put the answer in ourselves. It has to find hidden slack and delete it without changing any prediction; it has to leave an honest theory alone; and it has to refuse a patch whose new part moves no prediction — the property that stops “just add a parameter” from being free. 21 of 21 pass. This proves the code works, not that the idea is right.

Then — real physics, with nothing given away

A lattice field theory: a small grid carrying a quantum field with two flavours, of the kind used to study the strong force. It has both a genuine redundancy and genuine symmetries, and physicists know which is which — but we tell the machine nothing. It gets eight candidate transformations, most of which are not invariances at all, and has to sort them out for itself.

The one it deleted was the gauge redundancy. The four it kept were the flavour symmetries — and their conserved quantities really are conserved: they drift by about a part in ten billion, which is the accuracy of the simulation itself. The piece it deleted has no such conserved quantity, and the corresponding number drifts by 86%. Nine orders of magnitude separate the two branches, which is what tells you the sort is real and not a relabelling.

Then — can the language model find the missing piece?

Two worlds. In one, the data really is hiding a missing ingredient. In the other, the leftovers are pure noise and there is genuinely nothing to find. The model is not told which is which.

It proposes eight ideas either way, with the same confidence. The measurement is the only thing that tells the two worlds apart — which is exactly the division of labour we wanted, and exactly why the model is never allowed to be the judge.

Handed the true answer directly, the test recognises it easily — 97% against 20% for the wrong ones. So the weak link here is the proposer, not the test: the model kept suggesting near-misses and never once suggested the right one, even though the leftover signal was a clean wave.

Finally — a real dispute, where nobody knows the answer

Galaxy rotation curves: stars at the edge of a galaxy orbit far faster than the visible matter can explain. Two rival explanations have argued about this for forty years — invisible dark matter, or a modification to gravity itself. We pointed the machinery at real measurements of the galaxy NGC 3198 and asked which explanation carries something unmeasurable.

The answer is that it has nothing to say, and that is the useful result. Neither side is hiding anything unmeasurable. And the reason is worth understanding: our test compares two theories that agree on the data and asks which one carries dead weight. These two disagree on the data — one fits far better than the other. When theories disagree, you do not need any of this; you just compare them the ordinary way. The operation covers a narrower class of arguments than we assumed when we built it, and finding that out cost one real controversy.

Fourth — does any of it hold outside our own code?

Everything above was measured on systems we built, which is the obvious objection. So we ran the same two tests, with the thresholds imported unchanged, on 129 chaotic systems from a public benchmark we had nothing to do with. The reduction that throws away everything satisfies the first test on 96% of them, at a closure error of exactly zero. The second test refuses it on all 129.

A test that refuses everything would be no better, so we also built a suite where the right answer is fixed by construction rather than by the test. There the pair accepts all 24 reductions we know are present and none of the 72 empty ones — while the first test alone still accepts 47 of those 72. Handed the proposer's estimate of the right answer rather than the answer itself it accepts 0.71, so that shortfall is the proposer and not the test. It is not forgiving: tilt a correct reduction by 0.03–0.08 radians and it is refused.

And because the problem is about acceptance tests rather than about physics, we ran the same pair on two interpretability methods. Fitting an alignment between two language models on eight stimuli and scoring its R² in sample accepts 80% of the empty candidates. More data fixes that half of the test: held out on every position, R² alone accepts 10%. It does not fix the other half. Linear CKA gives a genuine correspondence and that same map composed with a random rotation the same score, because it never looks at the map being claimed, so the alignment test still accepts 60% of the empty candidates held out, and 80% on ESM-2 protein language models. Adding a causal test that perturbs one position and carries the change across refuses every empty candidate in every configuration while accepting the ones we know are real. On sparse-autoencoder features the correlational test refuses every empty control, yet of the features it calls interpretable an intervention supports 0.45, against 0.005 for matched controls. Where that intervention's own positive control is valid, it detects 0.90 of directions built to be causal. Passing the empty-input check is necessary, not sufficient.

Our own second test needed the same medicine. An early version preferred a vacuous candidate to the real one, 0.70 against 0.39, because overwriting the whole residual stream pushes both models toward the same common words. Localising the intervention removed that. We only found it by running the null.

Where this stops

The hard limit is not that reinterpretation is difficult. It is that some physics is not settled by measurement at all.

Everything above buys new concepts with observations: something is invisible to your instruments, so you act on it. That is one honest route to discovery. To find out how far it goes, we walked it through eight moments when the Standard Model of particle physics gained a genuinely new concept.

And the missing ingredient is not more data, or a cleverer search. You can only prove things about a theory if the machine has the theory in a form it can reason about — and no discovery system we know of carries one. That, rather than the difficulty of reinterpretation, is what actually bounds automated physics discovery today. It is also the most interesting open problem here.

The easy version of this shortcut is already dead. Guessing at new concepts from things that merely look wrong in a description was tested on three different problems and lost on all three. The difference is grade, not kind: a suspicious-looking description is a hint, whereas a proof says a thing is forbidden or it is not.

What you get

The corpus, the mechanism, and a script that checks our arithmetic.

Every number on this page is recomputed from the shipped data by scripts/summary.py, which exits non-zero if any published figure fails to reproduce. Run it before you trust us.

# all 30 published corpus statistics, recomputed from the data
python3 scripts/summary.py

# the mechanism
cd mechanism
python3 test_mechanism.py        # the 21 controls
python3 discover_symmetry.py     # L1: finds 5, sorts 1 from 4, emits 4 charges
python3 l2_slowfast.py           # L2: finds a level, refuses 3/3 empty ones
python3 l3_transfer.py           # L3: accepts the real one, refuses the control
python3 l4_posit.py              # L4: names the missing carrier
python3 live_rotation_curve.py   # where it goes quiet on a real controversy

Read this before using any of it

One annotator, and it is a language model.

Join us

This is an open project, and it is nowhere near finished.

Everything above is one group’s attempt at a problem that will need many. We would rather be corrected early than cited late, so the data, the code and both negative results are public and the failures are documented as carefully as the successes. Volunteers are welcome at any level — physicists, historians of science, machine-learning researchers, or anyone who just wants to argue with a label.

The single most useful thing: re-annotate a sample

We cannot tell from the inside whether our scheme is reproducible or whether it simply records one system’s habits. Only someone else applying it can settle that.

There is now a kit for exactly this, in corpus/reannotation/. It takes about an hour, needs no physics background beyond reading a prize citation, and runs nothing else of ours.

1. python3 make_sample.py draws 120 events with our labels withheld.
2. Read only the citation text. That is all our annotator saw.
3. Decide what the contribution changed, and how deeply. The definitions are in the kit’s README.
4. python3 score.py reports Cohen’s kappa against our labels. We publish the figure whatever it says — a low one tells us the ladder needs re-cutting, which is worth more to us than a confirmation.

Bigger things, if you want them