SRE-Bench is built around a single idea: keep the questions away from the model. Nineteen programs written from scratch by reverse-engineering specialists, more than five thousand expert hours (per the paper), averaging about 16,900 lines each, compiled into 262 binary instances with 44 hand-built anti-analysis mechanisms and 1,572 deterministically graded tasks. All of it private, all of it new, specifically so that no model could have seen any of it during training. That is an extraordinary amount of human labour spent on a property that has nothing to do with reverse engineering.
And OpenAI reports GPT-6 Astra at 99.2% pass@4, using its own harness and no step limit. On the Vals leaderboard, the same model scores 56.87%. Same model, different harness and a different metric, forty points apart. Hold that number, because it comes back in the section on acquired contamination.
It reads as a security story. Leaks, holdouts, secrecy, contamination. It is a measurement story, it is much older than any of this, and the discipline that owns it has had a name for it since before most of us were born. The name is item exposure.
An item that has been seen stops measuring
Here is the thing a benchmark actually is. Not a list of problems. A set of items, each of which exists to separate one examinee from another. An item earns its place by being hard enough that not everyone passes, easy enough that not everyone fails, and correlated with the thing you are trying to measure, so that whoever gets it right tends to be the one who is genuinely better.
That last property is called discrimination, and it is the one that dies first.
When an item circulates, its difficulty collapses, because the thing that made it hard was that you did not already know the answer. The item does not become wrong. It becomes uninformative. Everyone passes it, so it no longer separates anyone from anyone, and it contributes nothing to the score except a constant. A test made entirely of such items produces a number with no variance in it, which is a number that cannot rank, cannot detect improvement, and cannot fail anyone.
Where the curve crosses the line is the retirement date, whether or not anyone writes it down.
High-stakes testing has lived with this since the first time two candidates compared notes outside an exam hall. The answer was never to find the leak. The answer was to assume exposure as a constant, continuous cost of operating a test, and build the machinery to absorb it: large item banks, parallel forms assembled to the same specification, rotation schedules, and in adaptive testing, explicit exposure control that caps how often any one item can be administered. The method everyone still cites for that is Sympson and Hetter's, from the mid-eighties. Forty years of practice, and the whole point of it is that items wear out and you plan for it.
Every current fix is exposure control with a bigger budget
A taxonomy paper from August, by Angulo, Yeste and Espinos-Morato, organises contamination by the mitigation it defeats rather than by how it happened, which is the right way round. Five categories: direct, derivative, temporal, distributional, and acquired, with the last one occurring during the evaluation run itself rather than during training.
Line those up against what testing already does and the mapping is almost embarrassing.
Hiding the items closes the first row. The fifth row closes with the room.
Direct contamination is what a secure item bank is for. Derivative contamination, where a paraphrase or a derived variant leaked instead of the original, is what blueprint-based parallel forms are for: you assemble a new form to the same specification rather than reusing the same questions. Temporal contamination is item retirement on a schedule. Distributional contamination is the oldest one in the book and the least discussed here, because it is not about leakage at all; it is about whether the sample you built the test on resembles the population you claim to measure.
Which means SRE-Bench's five thousand expert hours were not spent on a novel problem. They were spent building one secure form, from scratch, by hand, because the field has no item bank to draw a form from and no blueprint to assemble one against. The cost is real and the work looks excellent. But it buys a single administration. When those 262 binaries are exposed, and they will be, the whole instrument has to be rebuilt from zero rather than refreshed, because nothing about how it was assembled was written down in a way that lets anyone assemble another one like it.
That is the part that should worry an operator more than the leak.
A secret instrument has no published validity evidence
Now the tradeoff nobody prices.
If I cannot see your items, I cannot check whether your benchmark measures what you say it measures. I can see a number and a leaderboard position. I cannot see whether the tasks sample the construct evenly or pile up in one corner of it, whether the hard items are hard for the reason you think, or whether two thirds of your score is being driven by one skill that happens to be easy to grade.
Testing solved this without giving up security, and the solution is the most transferable idea in this essay. You publish the blueprint and the statistics. You keep the items.
The blueprint is the specification: what the domains are, how many items come from each, what cognitive level each targets, what counts as a correct response. The statistics are the psychometric properties of the form: the distribution of difficulty, the distribution of discrimination, the reliability coefficient, the dimensionality. None of that lets you answer a single item. All of it lets an outsider judge whether the instrument is any good.
Nobody outside can answer a single item from the left column, but they can judge whether the whole instrument is any good.
Go and look at how many LLM benchmarks publish the left column. A task count and a description of the domain, usually. Difficulty distribution, almost never. Discrimination, essentially never. Reliability across runs, rarely, and when it appears it is a standard deviation over three seeds rather than anything you could call a coefficient.
So the field has arrived at the worst of both: instruments secret enough that nobody can audit them, and documented thinly enough that nobody can rebuild them. A private benchmark with a published blueprint is a serious measurement programme. A private benchmark without one is a number you have to take on trust.
Acquired contamination is a different failure
The fifth category in that taxonomy is the one I would separate from the other four entirely, because treating it as an item problem will send you to the wrong fix.
Acquired contamination happens during the evaluation run. The agent searches the live web and retrieves the answer, or reads the grading harness, or finds the repository the task was carved out of. The item was never exposed. The environment was.
That is not exposure control. That is administration conditions, which in testing is a completely separate apparatus with its own name: proctoring. You do not fix a candidate googling the answer by writing new questions. You fix it by controlling what the room contains.
The practical consequence for anyone running an agent eval is that network access during a run is a scoring condition, not an implementation detail, and it belongs in the report next to the score. The taxonomy paper lands on the same conclusion from the other direction: acquired contamination has to be documented with each reported score rather than once at benchmark release, because it is a property of the administration and not of the instrument. Two runs of the same benchmark, one sandboxed and one with a live browser, are two different tests. SRE-Bench's own numbers show the size of the gap: 99.2% pass@4 with a custom harness and no step limit, 56.87% on a public leaderboard. I cannot tell you from outside how much of that is the administration, and neither can the leaderboard, because neither run reports its conditions next to the score.
What this looks like on a Wednesday
Five things I would change in any eval programme that currently reports a leaderboard number.
Measure your own exposure instead of guessing at it. Keep a few items back from every release and re-run the retired ones each quarter. What you are watching is not the score, it is whether discrimination is falling. When an item stops separating your candidate models, it is spent, and you will see that long before you hear about a leak.
Write the blueprint before the items. Domains, how many items from each, what a correct response is, how it is scored. It costs an afternoon, it makes the next version assemblable instead of rebuildable, and it is the single artifact that lets someone else trust a suite they are not allowed to see.
Publish the statistics even when the items stay private. Difficulty distribution, discrimination distribution, reliability across repeated runs, and how many factors come out of the correlations. If that feels like too much ceremony for an internal eval, consider that you are using the output to decide which model handles your customers.
Treat a score with no interval as no score. One run is a point estimate of something noisy. I have written before about what the distribution behind a point estimate hides, and a benchmark number is the same argument in a different hat.
Budget item writing as a standing cost. Not a project, a line item. Items wear out, that is their nature, and an eval suite that was built once and never refreshed is measuring last year's models with last year's variance.
Close
The expensive part of SRE-Bench is not the reverse engineering. It is that five thousand expert hours bought one form, with no blueprint to assemble the next one and no published statistics to let anyone outside judge the first.
None of this requires new science. It requires noticing that an eval suite is a measurement instrument, that measurement instruments have a hundred years of operational practice behind them, and that almost all of the problems this field is discovering under new names were solved, slowly and expensively, by people trying to run a fair exam.
The leak is not the event. The leak is the weather. Build for it.