When a model solves a hard problem, the natural reaction is to say it understands the problem. My claim here is narrower: the evidence we collect from standard benchmarks is consistent with two very different stories, and we do not yet have reliable ways to tell them apart. One story is that large models develop semantic competence, abstractions, and reasoning. The other is that they learn an efficient compression of the training distribution and interpolate within that compressed space. Both predict the same thing on most of the tests we run.
Compression and interpolation, properly stated
The interpolation framing is often misstated. Balestriero, Pesenti, and LeCun (2021) show that in high dimensions, almost every test point lies outside the convex hull of the training set, so by the geometric definition, neural nets are almost always extrapolating in input space. And yet they generalize. The standard reading is that the relevant interpolation happens in a learned representation: training shapes a manifold on which novel instances of familiar patterns land in dense neighborhoods of training instances. Compression here means dimensionality reduction onto this manifold; interpolation means smooth behavior across it.
Figure 1. Test points (red) typically lie outside the convex hull of training data in input space, yet land in dense neighborhoods along the learned representation manifold. What looks like extrapolation by one geometry is interpolation by another. After Balestriero, Pesenti, and LeCun (2021).
This is enough to explain a lot. A model that has compressed millions of code samples can produce code for a request it has never seen, because that request projects onto a populated region of code-space. We call this "understanding code." The mechanical description does not need the word.
Why benchmarks cannot decide the question
Most benchmarks sample from a distribution close to training. Held-out splits in i.i.d. settings probe interpolation in the representation manifold, which is exactly where compression-based systems excel. Even tests advertised as "out of distribution" are usually only mildly so: different surface form, same underlying generative process. A model that has compressed the generative process well will look like it understands.
This is not a complaint about benchmark hygiene. It is a point about identification. If two hypotheses make the same predictions on the data we have, the data cannot distinguish them. To separate "good compression on a smooth manifold" from "abstraction with semantic content," we need probes those hypotheses disagree on.
Humans look different, with a caveat
Work on systematic and compositional generalization suggests that human concept formation has a structural property current models mostly lack. People can learn the meaning of a new symbol from one example and apply it correctly in novel combinations (Lake and Baroni, 2018, on SCAN-style tasks; later work on COGS shows transformers improving but remaining brittle). People also produce predictions that are stable under causal intervention: knowing that pushing the switch causes the light, not just that the two co-occur.
The caveat is real. Humans might be doing the same kind of computation as deep nets, with stronger inductive biases and better data efficiency. Compositionality might be a property of human representation manifolds, not a different mechanism. I do not think we know. What we can say is that whatever humans do produces a kind of invariance under distribution shift that current models reliably fail to reproduce, even at scale.
A proposal: understanding as out-of-distribution invariance
Here is the operational move. Stop asking whether a system "understands" in some essentialist sense. Ask whether its competence on a target abstraction is invariant under shifts that should be irrelevant to that abstraction.
A system understands addition if accuracy is stable as digit length grows beyond training range. Length generalization in transformers is mostly still broken without architectural intervention; positional-encoding tricks and scratchpads help but do not give robust invariance.
A system understands a causal relation if its predictions are stable under do-interventions on the cause, not only under observations. This is harder to test cleanly, but tractable in synthetic environments.
A system understands a compositional rule if it transfers to novel combinations of known parts at the rate predicted by the rule, not the rate predicted by training-set frequency.
The advantage of this framing is that it is falsifiable and graded. We do not have to decide whether a given model "really understands" multiplication. We measure the support over which its multiplication-behavior is invariant, and we report that support. "Understanding" becomes a quantity, not a property.
What this commits me to
I am not claiming current models cannot understand anything. I am claiming three more modest things. First, the standard battery of benchmarks cannot tell good interpolation apart from semantic competence, because the two hypotheses are observationally equivalent on those benchmarks. Second, the cognitive science evidence on humans gives us at least one sharper target: invariance under shifts on the relevant abstraction. Third, until a system reliably exhibits that invariance on hard OOD probes for a given concept, the parsimonious description of its competence is compression and interpolation on a manifold that happens to include this concept's neighborhood.
The direction worth pursuing is not to argue about the word, but to build systems and probes such that the answer stops being underdetermined. Mechanistic interpretability is one route; better OOD evaluations are another; causal benchmarks constructed with intervention semantics are a third. The myth worth retiring is the assumption that getting the right answer on familiar-shaped problems is evidence of understanding. It is evidence of competence. Whether competence is built from understanding or from very good compression is a question we have not yet learned to ask sharply.