That 90% Benchmark Score? The Model Did the Reading

Every few weeks, a new model launches with the same chart. A line going up and to the right, a pile of benchmark names you've seen a hundred times, and a press release breathlessly announcing 'state-of-the-art performance.'

Here's the thing nobody in the launch blog post wants to say out loud: the model has probably seen the test.

The Open-Book Exam Nobody Admits To

Pretraining corpora are scraped from the internet. The internet contains benchmark questions. Not just the datasets themselves — which get uploaded to GitHub and Hugging Face with the enthusiasm of someone leaking a movie — but also:

Blog posts walking through every GSM8K problem with 'helpful' step-by-step solutions
Stack Overflow answers that happen to match HumanEval tasks
Quizlet flashcards of MMLU questions
That one forum thread where someone pasted an entire eval suite because 'it might be useful'

Your model read all of that during training. Then it sat the exam. Then it scored 90% and everyone clapped.

This isn't intelligence. This is memorization with extra steps.

But We Deduplicated!

Cute. N-gram deduplication catches exact copies. It does not catch:

Paraphrases. Someone rewrote the question in their own words and posted it. Semantically identical, n-gram distinct.
Solutions explained in prose. 'The trick with the water jug problem is...' is the answer key wearing a trench coat.
Near-duplicates. Off-by-one numbers, swapped names, same skeleton.
The pattern, not the problem. Models learn that MMLU answers skew toward certain letters and that code evals follow common templates. You can farm a chunk of points without ever seeing the question.

A model that 'aces' a benchmark after seeing 40% contaminated data doesn't need the other 60%. It's seen the genre.

The Tell Is Always The Private Eval

Watch what happens when labs test their models on fresh, private benchmarks that didn't exist during training. The score magically drops. Not by a little. By a lot.

Sometimes the drop is honest contamination. Sometimes — and I say this with love — the training pipeline conveniently didn't clean the eval sets out of the crawl, because contaminated scores make better launch charts. Contamination isn't always an accident. Sometimes it's a roadmap item.

How To Check If Your Favorite Model Cheated

You can do this yourself in an afternoon:

Truncation test: Feed the model the first half of a benchmark question and ask it to continue. If it reproduces the exact question text and the answer verbatim, congratulations — it memorized the dataset. That's not reasoning, that's grep.
Canary strings: Some benchmarks embed canary strings specifically so labs can detect them in training data. Ask the model to complete the canary. If it does, someone did the reading. Almost nobody actually checks, which is why this keeps working.
Paraphrase the eval: Rewrite the questions yourself — new names, new numbers, same structure. Watch the score fall off a cliff. The GSM-Symbolic work made this painfully clear: swap a few nouns and numbers, accuracy tanks.
Compare public vs private scores: If a model beats everything on public evals and gets strangely quiet on fresh private ones, you have your answer.

What To Actually Trust

Fresh, rotating evals. If a benchmark has been public for two years, treat its score as decorative.
Deltas on brand-new benchmarks. When a new eval drops and model A beats model B on it the same month, that comparison means something. The same numbers two years later mean nothing.
Evals on your actual data. Your domain, your edge cases, your weird inputs. Boring, unscalable, and the only eval that maps to reality.
Human evals. Slow, expensive, annoying, resistant to memorization. That's why they're worth it.

A benchmark score measures how well a model knows the benchmark. That's it. That's the whole measurement. Every leaderboard built on a public, web-scraped test set is a curated archive of things the model was probably trained on, and every launch chart built on it is marketing wearing a lab coat.

The next time a model scores 96% on your favorite benchmark, ask one question: when did it do the reading?

Because it did. It always did.