The Failure Mode Humans Don't Have
Part 5 of Code in the AI-Primary Era. Previous: The Model Cannot List What It Knows.
A competent engineer’s ignorance is mapped. Ask what is not handled yet and you get a real answer: overflow is unchecked, the parser doesn’t do trailing commas, error paths are stubs. The map is imperfect, but it exists, and it is about the code rather than sampled from the same process that wrote it.
Models have no such map. When a model tells you an implementation is complete, that statement is generated by the same distribution, with the same coverage bias, as the implementation — a plausible completion, not a measurement. The previous part established why this is architecture rather than immaturity: a model cannot enumerate its own knowledge, so its omissions are unknown-unknowns to itself. This part is about what that does to a large artifact — a compiler, a database, a standard library — and the methodology it forces. The folklore alternatives (“the model knows this language, it just needs a reminder”; “a bigger model will close the gaps”; “ask it to double-check”) are each precisely wrong in ways the theory predicts.
One objection deserves to be met at the door, because the title invites it: humans have every one of these problems too, often with identical phenomenology. A human cannot enumerate their own knowledge either — recall is expensive and recognition cheap for us exactly as for models; a human’s bugs are unknown-unknowns to their author by definition, or they would not have shipped; and a human’s “looks done” is produced by the same head that produced the code, which is why independent review predates language models by half a century. If this part claimed that models fail where humans do not, it would be false on every point.
The claim is one level down. The failure modes match; the cognitive architectures underneath them do not — memory of a different volume and structure, attention with a different shape and budget, different input modalities, a differently built world model. The same failure mode running on different substrates fires on different inputs, in different places, at different times: a model’s blind spots and a human’s blind spots are drawn from different distributions. And weakly correlated error is the raw material of amplification — it is the entire reason a human–machine system can outperform both of its members, and it is what every remedy in this part actually engineers: checking the artifact from a distribution different from the one that produced it. Read the title in that light. What humans lack is not the ingredients of this failure mode; it is its manifestation in this particular texture — uniform, camouflaged, self-certifying — which is exactly what makes the human a useful check on it, and the model a useful check on ours.
Uniform shallow gaps #
Models are excellent at local-syntactic competence — code that is idiomatic, well-shaped, plausible line by line — and unreliable at global-semantic correctness — the artifact implementing its whole specification. On a large artifact, the resulting damage has a characteristic texture. It is not a few missing features. It is a roughly uniform distribution of shallow gaps: every feature present, every common path working, every tail silently absent.
The uniformity is what a generator optimized for the look of completion should produce. Finishing one feature exhaustively has diminishing plausibility-per-token — corner cases read as minor — while starting the next feature has high marginal plausibility. So generation goes breadth-first and shallow. A whole missing feature would look conspicuously incomplete; uniformly thin coverage looks done. Quantitatively: a specification is a conjunction of N constraints, an unconditioned generator satisfies each with probability p < 1, and the joint decays multiplicatively — a roughly constant fraction of constraints gets missed, smeared evenly across the artifact instead of concentrated anywhere you’d look.
Add the enumeration deficit and you get the failure mode humans don’t have: the gaps are invisible to the model, invisible in its confident self-assessment, and invisible to anyone who reviews only what is present rather than what is absent. And the reflex countermeasure — have the model write its own tests — actively hides it. A self-authored suite samples the same compressed feature-model as the implementation, exercises exactly the implemented subset, and passes. Green, and meaningless on precisely the question it seems to answer.
The external corpus is not a convenience #
If completeness cannot come from inside the model, it must be driven from an independent extent whose coverage does not share the model’s bias:
Completeness must be checked from a distribution different from the one that produced the code — an imported test corpus, a reference implementation, a formal specification.
For Logos, whose surface descends from Rust, that means importing the rustc/core/alloc/std test corpora and running them against the AI-built implementation. It is worth being precise about what this is. It is not extra rigor bolted on for comfort. The corpus is the materialized, scannable extent the model lacks — an index bolted onto a point-query engine. And unlike offloading arithmetic, what is offloaded here is not a computation but content: the externalized ledger, accumulated by the reference’s authors over decades, of everything that must hold. The alternative — surfacing tail omissions by sampling the model — is exponential exactly where the gaps live.
Reading the discovery dynamics #
Run an external corpus against an AI-built artifact and the discovery process produces a characteristic signal. The obvious readings of it are wrong, and the wrong readings are expensive.
A constant discovery rate means sampling, not draining. New gaps surface at a roughly steady rate per unit of corpus consumed — confirming the uniform-gap picture. The rate tracks the corpus’s coverage growth, not the gap population’s depletion, so it says almost nothing about how close to done you are. Answering that requires a coverage measure over the requirement space — an external structural prior the rate itself does not contain.
Gap depth falls while the rate holds. Deep structural absences — whole subsystems — are high-traffic and get hit early. As the implemented extent grows, new corpus cases land increasingly in its interior, and closing them becomes local work. Late discoveries stop looking like “gaps” and start looking like ordinary bugs: mechanism present, edge wrong. That shift is real progress — the frontier between never-learned and known-but-unemitted receding — even while the raw discovery rate stays flat.
Beware the false summit. Eventually the rate drops, and the dangerous reading is “done.” The likely truth is that you exhausted the corpus, not the gaps. What remains lives in the corpus’s own blind spots — the cases its authors didn’t think to test — and it resurfaces in production, because corpus coverage and usage coverage are different distributions. The difference between them is, quite exactly, the set of gaps that ship.
Instance coverage is not class coverage #
There is a second trap on the repair side. Shown a failing test, a model makes the minimal local fix — the generation/verification asymmetry again: the fix is conditioned on one instance, and generalizing to the instance’s class would require enumerating the class. So a green corpus can be a stack of point-fixes, each satisfying one imported test, none covering the class the test was sampled from. The tell is diagnostic: if newly discovered gaps increasingly recur in already-touched classes, your fixes are pointwise.
The remedy is a generalizing pass: for each passing imported test, identify its problem class by name, generate sibling cases — metamorphic variants, fuzzing around the same construction — and adjudicate them against the reference oracle. A class that passes only its one imported instance was point-fixed; a high sibling pass-rate is genuine generality. Honest caveat: the pass rides on an external class taxonomy, so you generalize only within classes you named, and the residual risk moves up into unnamed classes. The closed-world boundary rises a level rather than disappearing. A level is a lot.
The model, correctly placed #
None of this demotes the model to a repair tool. Its correct role in the loop is the proposal distribution: it makes oracle-driven search tractable by proposing good candidates — implementations, sibling tests, fixes — at negligible cost, turning blind enumeration into guided exploration. The oracle adjudicates; verified results materialize into a growing external extent; the growing extent expands what the model handles as routine, so the novelty frontier recedes. Neither half closes the loop alone: the corpus without the model is a list of failures nobody converts into code; the model without the corpus is a generator of confident, uniformly incomplete artifacts.
Logos builds this loop in as infrastructure rather than heroics: the imported reference corpus is a managed build-system input in lforge (importing, trimming, re-importing as gaps close is a first-class workflow); the gap ledger — known-missing behaviors keyed to the requirement space — is a durable project artifact rather than something that evaporates with a conversation; differential runs against a reference implementation are a cheap primitive; and the platform’s own data — diagnostics, registries, provenance, the ledger itself — lives in Writ, a structured, navigable, invertible store. The design brief in one sentence: supply, as ordinary infrastructure, exactly the index the model cannot be.
Next: One System, One Dynamics — the human’s actual role in the loop: responsibility, the ownership spectrum, and the two goals an AI-first platform must serve.