Newsletter

November 2026 | Issue 9

AI: Answer Keys

Every leaderboard you have ever trusted grades a model against an answer key, and nobody checks the key. When researchers finally did, one in sixteen MMLU questions turned out to be broken, and more than half of its virology section. The AI field is now rediscovering, one embarrassing audit at a time, what exam writers, survey designers and software testers learned decades ago: the question is not the hard part. The key is. So we built a lab that proves the key before a model writes a single word.

The simple picture

A benchmark is a list of questions with an answer next to each one. A model reads the question, writes an answer, and a script compares the two. Score goes up, press release goes out, a startup raises a round. The whole machine assumes one thing it never says out loud: that the answer next to each question is right.

It often is not. When Aryo Gema and colleagues re-read 5,700 MMLU questions by hand in 2025, they estimated that 6.5 percent of the whole benchmark was flawed: answer letters shifted during scraping, questions with two right options, questions with none. In virology the flaw rate was 57 percent. Joshua Vendrow and colleagues went through every GSM8K test problem that a frontier model got wrong and found 120 of the 1,319 that were ambiguous, inconsistent or miskeyed. OpenAI paid 93 developers to audit the scraped tasks behind SWE-bench and threw out more than two thirds of them for vague instructions or unit tests that rejected correct fixes.

These are not obscure datasets. They are the scoreboards the industry uses to decide which model is smart.

Noise is not a headwind

The comfortable assumption is that a few percent of bad keys is like a strong wind: it slows every runner by the same amount, so the race still finishes in the right order. The audits say otherwise. On the original GSM8K, Claude 3.7 Sonnet and Llama 405B tied at 45 errors each. On the cleaned version, one made 2 errors and the other made 17. An eightfold gap in reliability had been hiding behind 120 broken questions.

The same pattern shows up everywhere someone looks. When Gema's team re-scored MMLU's virology section on clean questions only, the model in first place fell to third and a fourth-place model rose to first. When EvalPlus gave HumanEval eighty times more unit tests, every model's score fell and the rankings reshuffled. Broken keys do not add noise evenly. They reward whichever model happens to share the key-writer's mistakes, and they punish the one that read the question carefully.

That last point is older than it sounds, and we will come back to it.

Try it The tie that wasn't

Two frontier models scored exactly the same on GSM8K. Fix the 120 broken questions and see what the tie was hiding. Error counts from GSM8K-Platinum (Vendrow et al., 2025).

Nine judges, two votes

The fashionable fix is to stop writing keys at all and ask a language model to grade the answers. It is cheap, it scales, and it produced the most quoted number in evaluation: GPT-4 agreed with human raters about 85 percent of the time on open-ended chat, setting ties aside. The trouble is what that number measured. Chat preferences are mostly taste. When the task changes to checking whether an answer is actually correct, the judges fall apart. On JudgeBench, a set of hard, objectively gradable pairs, a vanilla GPT-4o judge scored just under 51 percent, which is a coin.

The instinct after that is to hire a jury: ask several models from different companies and take the majority. It helps less than you would think. Guneet Kohli's study this year ran nine judges from seven model families against a hundred human annotators and measured how often their mistakes lined up. They line up a lot, because the models read much of the same internet. Nine judges behaved like 2.18 independent votes. An infinite panel at that level of agreement tops out at about two and a half. Our own lab uses two readers from different families to certify its word lists, and by that arithmetic two readers are worth about one and a half votes, not two. We call that evidence, not proof, and now there is a number behind the word.

Try it How many votes is your jury worth?

Add judges and watch the independent votes they are worth. The overlap starts at 0.391, the average correlation between judges' mistakes that Kohli measured across nine judges from seven families. Drag it to zero to see what independent judges would buy you.

Everyone else already paid for this lesson

In 2008, Marie Tarrant and James Ware audited ten high-stakes nursing exams and found that nearly half the questions broke a basic rule of item writing: a giveaway word, an option much longer than the rest, two answers that could both be right. When they removed the flawed questions, the pass rate fell, but the share of top students scoring above 80 percent rose. Bad questions had been carrying borderline students over the pass line and holding the strongest ones down. That is the GSM8K result, forty years early, with nurses instead of models.

Survey designers learned the wording half of the lesson even earlier. In 1940, Americans were asked whether the country should forbid public speeches against democracy, or whether it should not allow them. The two questions mean the same thing. The answers differed by twenty-one points. Software testers learned the coverage half: random test generators are powerful, but QuickCheck's own authors warned in 2000 that nothing tells you which cases you never generated. And medicine learned the planning half. Once large heart and lung trials had to register their outcomes before the results came in, the share reporting a win dropped from 17 of 30 to 2 of 25. Psychology's registered reports tell the same story: positive results fell from 96 percent to 44.

The AI field is re-deriving all four lessons in public, one benchmark audit at a time.

Prove first, word second

So we built a lab around the order of operations. It is called reasoning-dataset-lab, rdlab for short, it is open source, and it builds puzzles, quizzes and verification items backwards from the usual way. Code constructs a hidden state first, a small world with facts in it. A solver proves the answer key from that state. Only then does a local language model put the item into words. A key that a model chose is an opinion. A key that code built from a hidden state and a solver proved is a fact about the item.

The second idea comes straight from the medical trial registries. Before any item exists, the dataset is written down as a map: every kind of item we intend to build, how many, how varied, and how much of it the generator can actually produce today. A cell the generator cannot build yet stays on the map at capacity zero with a reason written next to it. The easy move, which most synthetic datasets make without meaning to, is to quietly drop the cases the generator handles badly and publish what is left. That makes the benchmark look complete and makes every model look better on the domain than it is. A capacity-zero cell keeps the hole visible. One hash covers the map and the project files, so every build can say exactly what it was built from.

The third idea is the sealed contest. When we needed to pick which small local model should judge claims, we built the test set the same way, keys proven by code, and then sealed them. A probe throws the whole set out if its answers can be guessed from surface wording alone. Five candidate judges ran on rented machines without ever seeing a key. Only after all five finished did the keys come out and meet the answers, and the winner was picked by a rule written down before anyone looked.

Where it still leaks

A solver proves the answer to the world you wrote down, not the world. If the rules you encode are wrong, the key is proven wrong with perfect rigour. Formal methods people have a name for this, the specification bottleneck, and we hit it ourselves. Our lab swaps values within lists, like a pump's operating modes, to make claims false, so every pair in a list has to be a real contradiction. In our October certification run, 44 of 50 lists passed. Of the eleven pairs that failed, six were genuine mistakes in our own lists that the first check had missed. A turbine can be accelerating and online at the same time while it ramps up load. Our code had said it could not.

The second leak is the words. The key is proven on the hidden state, but a language model writes the sentence the reader sees, and small models are sloppy narrators. In outside studies of open models turning structured data into text, three quarters or more of the outputs contained at least one factual slip. We have not yet measured how often our rendered items drift from their keys, and that is the honest gap. Until we have that number, nothing the lab writes counts as training data until a person has read it, and the person reads the question before seeing the key, because people trust fluent machine text more than they should.

Moves for Monday

If you ship a product on top of a benchmark score, read twenty of the benchmark's questions yourself before you trust it. It takes half an hour, and the audits above suggest you will find at least one you would argue with.

If you use a model to grade your outputs, swap the order of every pair you show it and count how often the verdict flips. If it flips more than a few percent of the time, you are measuring the seating chart, not the answers.

If you use a panel of judges, do not count them as independent. Two models from different companies are worth about one and a half votes, not two. Spend the difference on a person.

If you build your own evaluation set, write the coverage plan down before you write a single question, and leave the cells you cannot fill on the plan. The holes are information.

And wherever you can compute the answer instead of asking for it, compute it. Code is a terrible narrator and an excellent key.

Looking ahead

Two predictions, dated so they can be graded. First, by the end of 2027 at least one of the five most-cited language model benchmarks will publish an official corrected edition with its error rate stated, the way MMLU-Redux and GSM8K-Platinum did unofficially. Second, the model-judge panels that every evaluation platform now sells will start reporting effective votes instead of judge counts, because the correlation numbers are now public and somebody's customer will ask.

Last issue's six bets are still open; the first one, an AI-made game from xAI, settles on New Year's Eve. We will grade it here.

The lesson underneath all of this is boring and old: the test is only as good as its answer key, and the key deserves more scrutiny than the student. Exam boards, pollsters and drug regulators all learned it the expensive way. The rest of us can just read their notes.