Most eval suites I read are written like unit tests. Assert that the output matches, count the greens, ship on the number. The frame is wrong in a way that costs real money, because a unit test observes the thing it is testing and an eval does not.
You cannot see whether a model reasons. You see two hundred responses and infer something you never observe directly. That inference is the whole discipline of psychometrics, a field that spent the twentieth century working out how to measure things nobody can point at: intelligence, depression, reading ability. It produced a set of results about when such measurement works and when it quietly fails.
The AI industry is rediscovering those failures one blog post at a time. Benchmarks that saturate. Scores that move without the model changing. Judges that agree with humans on average and disagree in a pattern. None of this is new. It has names, and the names come with fixes.
A score without a standard error is not a score
Run your suite five times against the same model and the same prompts. The number moves. Sampling temperature, the judge's own sampling, ordering effects inside a context window, a retrieval step that returns a different neighbour today. Every one of those is measurement error, and reporting 87.3% from a single run reports one draw from a distribution as if it were the distribution.
Classical test theory writes this as observed score equals true score plus error, which sounds trivial until you take the consequence seriously: reliability is the ceiling on validity. A suite that disagrees with itself across runs cannot agree with anything else either, including the production outcome you actually care about. Correlations get attenuated by unreliability, and no amount of clever analysis downstream recovers what the noise ate.
The left panel is what gets pasted into the release notes. The right panel is what the same runs actually support.
The practical rule is cheap. Run the suite n times, report the mean and the spread, and treat any improvement that sits inside the band as no improvement. I would rather see 87.3 ± 2.1 over five runs than a confident 89.1 from one, because the first tells me whether next week's number means anything.
This also settles the most common argument in an AI team, the one about whether a prompt change helped. Usually the honest answer is that the suite cannot tell, and knowing that is worth more than the change.
One number that is several things
The second failure is structural. A suite that mixes tool-call correctness, schema compliance, refusal behaviour and multi-step reasoning produces a composite score, and when a composite moves you cannot say what moved.
Measurement calls this dimensionality, and it is not a philosophical question, it is an empirical one you can answer in an afternoon. Score every item across the model versions you have. Correlate the items. If they cluster, your one number is several numbers wearing a coat, and the honest report is subscales.
The left structure is what a single headline number assumes. The right one is usually what the correlations say.
I spent years on exactly this problem in another field. Dimensionality assessment in bifactor structures, which is the case where a general factor and several specific factors coexist and the naive analysis attributes everything to the general one. The failure mode transfers without modification: a model that got better at emitting valid JSON raises a suite average that a reader will interpret as better reasoning, and nobody in the room is lying.
Report the subscales. If the decision you are making depends on one of them, weight it explicitly and write down why, rather than letting the arithmetic of an unweighted average make the choice for you.
Most of your eval set carries no information
Item analysis is the oldest tool in the drawer and the one that pays fastest. Every item has a difficulty, the share of models that pass it, and a discrimination, how well it separates the models you are choosing between.
An item that every model passes and an item that every model fails both carry zero information about the ordering. They cost tokens on every run, they inflate or depress the headline, and they contribute nothing to the decision. In a suite that has been around for a year, they are usually the majority.
Saturation is not models running out of headroom. It is items running out of discrimination.
This is also the real explanation for benchmark saturation, and it is less dramatic than the usual telling. A benchmark does not stop working because models got too good in some general sense. It stops working because its items drift to a difficulty where nothing separates, and a test with no variance left has no signal left. That is a property of the instrument, not a fact about intelligence.
An LLM judge is a rater, and raters get audited
Swapping a human panel for a model judge is not a cost optimization, it is a change of instrument, and it carries the obligations that come with one.
A rater owes you agreement with a criterion, and agreement means a coefficient that corrects for chance rather than a raw percentage. Two raters marking 90% of a mostly-passing set identically can be agreeing almost entirely by accident. It owes you stability, because the judge is a model and models get updated underneath you. And it owes you an answer about differential behaviour: a judge that agrees with humans 90% of the time overall, and disagrees systematically on one population of candidates, has an average that hides the only result that matters.
That last one stops being an academic concern the moment the system touches hiring. It is also, not coincidentally, what an auditor will ask for, which is the point where the measurement argument and the regulatory one turn out to be the same argument.
Validity is something you argue, not something you have
The last piece is the one with no shortcut. Validity is not a property a suite possesses. It is a claim that the score supports a specific decision, and the claim has to be made and defended.
The version that bites in production is predictive. When the decision is downstream and slow, a hire, a resolved ticket, a renewal, then face validity is worth nothing and the only real evidence is whether today's score predicts that outcome months from now. Which means building the link deliberately: log the eval score against the decision, wait, and check. Most teams never close that loop and then wonder why an offline win never showed up in the business.
Write the argument down in two sentences. This suite measures X. We use it to decide Y, and here is the evidence that X predicts Y. If you cannot write the second sentence, you have a dashboard rather than an instrument.
What this looks like on a Wednesday
The stack is ordinary. Promptfoo for the suites, Langfuse for traces, Datadog for what happens after, self-hosted Qwen on AWS via vLLM for the privacy-sensitive paths and frontier APIs for the rest. None of that is the interesting part.
The interesting part is the four questions I ask before a number is allowed to justify a decision. What is the spread across runs. Is this one construct or three. Which items still discriminate, and what percentage of the suite is dead weight. And if a judge produced the score, what is its agreement with a human criterion, computed this quarter rather than the one it was validated in.
They take an afternoon between them. They are the questions a psychometrician asks of any new test before anyone is allowed to decide anything with it.
Closing
There is a version of this field where everyone rebuilds measurement theory from scratch, badly, over the next five years, and publishes the results as discoveries. It is already happening, and the tell is how often a post announces something that has a name in a 1951 paper.
The alternative is cheaper. The instruments are old, the failure modes are catalogued, and the fixes are mostly arithmetic you can run this week. Your eval suite is a measurement instrument whether or not you treat it as one. The only choice is whether you find out what it measures before or after it decides something.