Nineteen of twenty patched the symptom

In the previous article I described a habit I kept running into: when a system has a language model inside it and something goes wrong, the explanation drifts towards the model. The sampling. The nondeterminism. Something nobody wrote and therefore nobody has to fix.

That was an observation from work, which is a polite way of saying it was an anecdote. So I built a way to measure it, and the measuring turned out to be harder and more interesting than the result.

The setup

I built a small classifier that reads a repository's documentation and assigns it a category from a hierarchy, with a confidence and a justification. Real repositories — fifty of them, low-star and recent so that no model has them memorised. It runs on a real model. It has a search budget, a context window that truncates, a retrieval step, and a trace.

Then I planted a fault in it, ran the same batch five times, and got a table where a handful of projects change category between passes even though nothing changed between passes.

That table is what the agent sees. The only thing I varied is whether it also gets the code.

The whole design. Two groups see the identical failure; one of them can open the system. Nothing else differs, including the prompt asking them to investigate.

Scenarios were frozen with a hash before any of this ran, the people coding the answers never knew which group a response came from, and the statistics were computed from the raw JSON by a script that ships with the repo. I'll come back to why all that ceremony mattered.

The first fault: the order of retrieval

The planted fault: the search that retrieves documentation resolves score ties using an identifier derived from the request, so each pass hands the model a different set of documents. The variability comes entirely from the input. The model's sampling has nothing to do with it.

Twenty agents saw the table without the code, twenty with it.

The headline isn't the attribution. Blaming the model outright happened four times out of twenty, and at p = 0,053 that doesn't clear the usual bar. The strong formulation of my own thesis — the agent blames the model — did not survive contact with the data, and I'd rather say so plainly than round it into significance.

What did survive is the behaviour. Nineteen of twenty, with no way to see the system, set about damping its output — voting across retries, pinning the temperature, averaging the instability away. With the code in front of them, nine. You don't have to say the model is at fault to treat it as if it were: it's enough to stop looking for a cause and start smoothing the symptom.

The second fault: where blaming the model is half right

The first scenario has a weakness I could see from the start. Nothing in it makes sampling a reasonable explanation — it's just the lazy one. A fair test needs a case where a competent engineer could reach that conclusion in good faith and still be wrong.

So I built a second one, and this is the part I'd defend hardest.

A cache shared across the runs of one batch, keyed by project and section. Two unrelated projects that happen to share a short name collide, and the second one to ask receives the first one's documentation. The model then classifies that documentation perfectly well — it is simply not the documentation of the project it was asked about.

Why this one is fair: the symptom is "it fails in the batch, and reproduces fine on its own", which is the canonical signature of nondeterminism. The affected projects wander from pass to pass. The justifications read like textbook hallucination — fluent, confident, describing features the project doesn't have.

And here's the part that makes it honest. Of the fifteen label changes in the table, fourteen are contamination and one is genuine model sampling — a project that was never served anyone else's documents and moves anyway, confirmed by resampling its prompt twenty times. For that one, "it's the model" is the correct answer.

So the rubric has two opposite fields that can both be true of the same response: blaming sampling for the fourteen is the error, and attributing the one correctly is the win.

Making the wrong explanation reasonable is what moved the number: from 4/20 at p = 0,053 to 6/20 at p = 0,0101. The reflex isn't summoned by laziness. It's summoned by a situation where it half fits.

One result went the opposite way from what I expected: proposing to fix the temperature was more common with the code than without (8/20 vs 2/20). Reading the responses explains it — with the source in hand they can see the residual sampling is real, and they propose pinning it as part of a fix. It's an informed suggestion, not a reflex, and it's a good reminder that a single checkbox measures behaviour badly.

Nobody asked. A hundred and forty times.

Across every scenario and every group, not one response asked for the information it was missing before reaching a conclusion.

Not once. Every package deliberately shipped shallow logs — input and final answer only — while the full trace existed and would have been handed over on request. The brief said so: you can ask for whatever you're missing. All of them diagnosed first and listed what they'd lacked at the end, after the conclusion was already written.

The obvious explanation is social: being shown a table of outputs frames the job as analyse this, and asking for more input reads as refusing the job. That's testable with one sentence, so I tested it. Sixty more responses, same frozen scenario, with the brief rewritten to say that asking is a complete answer, preferable to a hypothesis you can't check — and closing with asking is not leaving the job half done.

A hundred and forty out of a hundred and forty. Telling them in as many words that asking is not shirking changes nothing about whether they ask.

What it does change is the shape of the answer. Hedging doubles. Given explicit permission to ask, they don't ask — they cover themselves, flagging the conclusion as provisional instead of doing the one thing that would make it firm.

And the other half of the explanation is what they do instead. They build their own measurement: a script, a sweep over the corpus, a synthetic reproduction of the pipeline. On the stronger tier, 18 of 20 do this, against 7 of 20 on the weaker one (p = 0,0004) — the largest capability difference in the whole study, and it isn't about willingness to ask, which is zero everywhere. It's about being able to manufacture the answer without asking.