The Model Cannot List What It Knows
Part 4 of Code in the AI-Primary Era. Previous: Attraction Basins.
Try a small experiment with any frontier model. Pick a language feature it knows well and ask about one specific edge case — it answers correctly, immediately, in detail. Now ask it to list all the edge cases of that feature. You get a list: fluent, plausible, incomplete — and incomplete in a way the model cannot detect, because the items it omitted are omitted from its self-report too.
The difference between those two queries — evaluating at a point versus enumerating a domain — is, I think, the most underrated fact about language models in engineering practice. The previous part treated the model as a dynamical system and asked which basin a trajectory lands in. This part asks the static question: what is stored in the weights, and which operations does that store actually support?
A compressed, forward-only program #
Generalization is compression. Training does not build a table of the corpus; it distills the corpus’s regularities into a short program — that is why a model generalizes at all, and it is the memory-side view of the compressibility axis. But the compression is implicit. Nowhere in the weights is there a materialized inventory of what was learned. There is only a function: give it an input, it evaluates a learned map at that point.
This makes the model a point-query engine. It computes at a point, cheaply, in one forward pass. Two operations are architecturally absent:
- Inversion — “which inputs produce this output?” There is no preimage operator.
- Enumeration — “list everything you know of kind X.” There is no domain scan.
Both would require a materialized, navigable extent — the induced partition of input space written down somewhere walkable. It never is. It is evaluated, point by point, on demand, and discarded. A database exposes a walkable index; a model exposes a callable function. This is not a capability gap that the next scale-up closes; it is what “storing knowledge as a compressed forward function” means.
Two kinds of gap #
Now look at what this does to generated artifacts. When a model emits an incomplete implementation — common cases handled, remainder silently absent — the omissions come in two structurally different kinds:
The OoD gap. The case was never learned. No mass in the training distribution, no bridge from generalization; the learned function simply has nothing there. Closing it requires acquiring the knowledge from outside — documentation, a reference implementation, a failing test with the answer in it.
The InD gap. The knowledge is present. Present the case explicitly and the model handles it immediately and correctly. It was omitted from the artifact because emitting it would have required enumerating the feature’s sub-behaviors — the exact operation the architecture does not have. The model knows it, and cannot list it.
The InD gap is the expensive one, and the strange one — a deficit of access, not of knowledge. And before you probe, the two kinds are externally indistinguishable: both just read as “X is silently absent from the output.” Which kind you are looking at is determined only by experiment: show the model the case and see whether competence appears.
This is the same fact as the archipelago, stated statically. “Basins nucleate on InD islands, and ordinary prompts contract to the modal basin” is the dynamical sentence; “mass exists in the tail, but forward generation never visits it” is the static one. Models must be led through the basin landscape and the list must come from outside are one proposition in two vocabularies.
Generation is not verification #
Why does generation skip things the model knows? Because producing a complete artifact and checking a flagged case are different information problems.
Generation means selecting one trajectory out of an exponentially branching space while allocating effort across thousands of sub-behaviors — with no internal salience signal saying which corner cases exist and matter. Under that pressure, coverage comes out uniformly shallow. Verification of a specific case is the opposite: the failing example pins the behavior, the surrounding code pins the structure, and the conditional entropy of the answer given all that is small. The model is fluent exactly there. Recognizing is cheap; finding is expensive — for models more than for anyone, because a model cannot even enumerate the space it is supposed to be searching.
Two corollaries follow that sound paradoxical until you have this asymmetry, and then become obvious:
Self-written tests inherit the artifact’s blind spots. A model-authored test suite is drawn from the same compressed feature-model that produced the implementation. It exercises the implemented subset — thoroughly, even — and the gaps sit in that model’s complement, which the suite cannot reach from inside. The suite passes. The pass means: the code agrees with itself.
Self-checking does not surface InD gaps. “Review your code and check it is complete” samples a completeness assessment from the same distribution, with the same coverage bias, as the code. “Looks done” is one more plausible completion. The model’s confidence indexes basin membership — not correctness, and not coverage.
No way out from inside #
It is tempting to think clever sampling escapes this. Ask the model a thousand times, at high temperature, from many angles — eventually everything it knows falls out, doesn’t it?
In the limit, yes; in practice, no. InD gaps live in the low-probability tail — that is why they went unemitted. Surfacing an item of emission probability ε costs on the order of 1/ε samples, and the gaps span exponentially many tail regions. Sampling-to-enumerate is brute-force inversion — re-paying, at generation prices, the exponential cost that the missing inverse operator was hiding. It is cheap exactly where it is useless (the modal behaviors you already have) and astronomical where you need it (the tail), with no stopping rule and no completeness certificate at any point.
Nor is this deficit offloadable in the way arithmetic is. Offloading works when the function’s domain is external: numbers can be handed to a calculator, a query to a database. “Enumerate what this model knows” has the model itself as its domain. A function over X cannot be delegated to a party without access to X — and the only party with access to the learned extent is the model, which is precisely the architecture that cannot scan it. Deficit and data are locked in the same place. Arithmetic is a missing capability, and capabilities can be outsourced; enumeration of self is a missing reflexive operation, and it cannot be.
So the conclusion stands on architecture, not on pessimism about any particular model:
The model cannot enumerate its own knowledge, and no amount of self-querying induces it. If it cannot list what it lacks, the list must come from outside.
What that external list looks like for real codebases — reference corpora, gap ledgers, differential oracles, and the characteristic rhythm of discovering thousands of silent omissions in an AI-built artifact — is the next part, and it is where this series stops being theory and starts being methodology.
Next: The Failure Mode Humans Don’t Have — uniform shallow gaps, the mandatory external corpus, and the false summit.