I Checked 23 Model Launches for Checkable Benchmarks. Zero Published a Single Model Output.

TL;DR: No, the benchmarks are not broken. The marketing is unfalsifiable. On 2026-08-15 I read the official launch announcement, model card or technical report for 23 frontier and near-frontier model releases across 15 vendors. 22 of 23 report benchmark scores. 12 of 23 name the eval harness or link a reproducible config. 5 of 23 say anything about training-data contamination. 0 of 23 publish the raw per-task outputs, which means not one of those 22 score tables can be independently checked at the item level.

Every launch leads with a table. Nobody argues the tables are perfect. The narrower, answerable question: if you wanted to verify one of those numbers, could you? So I stopped reading commentary about benchmarks and read the launch pages. (The leaderboards built on top of them disagree just as loudly: seven public boards named six different number ones on one day.)

What exactly did I check?

For each release I fetched the primary document the vendor published and scored five things: (a) are benchmark scores reported, (b) is a specific eval harness named or a reproducible config linked, (c) is contamination or leakage mentioned, (d) is the prompting and test-time setup stated, (e) are raw per-task outputs published rather than aggregates.

I was deliberately generous. Where a vendor published a technical report or a separate evals-methodology document, I scored that instead of the announcement. Partial disclosure counted as yes.

Not everything resolved cleanly. openai.com refused my fetcher, so I retrieved those three pages directly. Amazon's Nova 2 report 404s at assets.amazon.science and serves fine at cdn.amazon.science. qwen.ai/blog rendered seven characters of text, so Alibaba is represented by its Hugging Face card.

Release (doc read)	Harness	Contam.	Setup	Outputs
OpenAI GPT-5.2 (Dec 2025)	N	N	Y	N
OpenAI GPT-5.5 (Apr 2026)	N	Y	Y	N
OpenAI GPT-5.6 (Jul 2026)	Y	N	Y	N
Anthropic Claude Opus 4.7 (Apr 2026)	Y	Y	Y	N
Anthropic Claude Opus 4.8 (May 2026)	Y	N	Y	N
Google Gemini 3.1 Pro (evals doc, Feb 2026)	Y	N	Y	N
xAI Grok 4 (Jul 2025)	N	N	Y	N
xAI Grok 4.6 (Aug 2026)	N	N	N	N
DeepSeek V4-Flash (model card)	N	N	Y	N
Meta Llama 4 (Apr 2025)	N	N	N	N
Mistral 3 / Large 3 (Dec 2025)	N	N	N	N
Alibaba Qwen3.6-27B (model card)	Y	N	Y	N
Amazon Nova 2 (tech report, Dec 2025)	Y	N	Y	N
Microsoft MAI-Thinking-1 (Jun 2026)	N	N	N	N
Cohere Command A (tech report)	Y	Y	Y	N
AI21 Jamba 1.6 (Mar 2025)	N	N	N	N
Moonshot Kimi K2 Thinking (model card)	Y	Y	Y	N
Moonshot Kimi K3 (model card)	Y	N	Y	N
Z.ai GLM-5.2 (model card, Jun 2026)	Y	N	Y	N
NVIDIA Nemotron 3 Ultra (model card)	Y	N	Y	N
IBM Granite 4.1 (model card)	N	N	Y	N
Ai2 Olmo 3 (Nov 2025)	Y	Y	N	N
MiniMax M2.7 (model card)	N	N	N	N

What did the count show?

The zero in the last column is the finding. Twenty-two documents publish score tables. None publish the per-item results those tables aggregate. You cannot audit a single answer.

The one document reporting no benchmark number at all is Microsoft's MAI announcement, which calls MAI-Thinking-1 "competitive with models in its weight class on SWE-Bench Pro at substantially lower cost" and never gives the score. That is the end state of an uncheckable claim: eventually you stop bothering with the number.

The worst-documented tables are not from small labs. xAI's Grok 4.6 page posts ten benchmarks with version numbers and no methodology note. Its only caption explains where the competitors' numbers came from: "Competitor figures are drawn from the respective developers' published system cards or benchmark leaderboards." The comparison is xAI's own runs against numbers xAI did not reproduce.

Who actually showed their work?

More vendors than I expected.

Google's Gemini 3.1 Pro evaluation document is the best of the 23. It states "All Gemini scores are pass @1 except where otherwise noted" and that "'Single attempt' settings allow no majority voting or parallel test-time compute," names the Terminus 2 harness, reports results "Averaged over 10x runs for SWE-Bench Verified," and releases the MRCR v2 dataset on GitHub.

Moonshot's Kimi K2 Thinking card does the most useful thing in the sample. It reports that web access to Hugging Face leaks benchmark answers, that the model "can achieve a score of 51.3 on HLE without blocking Hugging Face," and that they blocked it anyway. They published the inflated number and the one they stand behind, giving away roughly six points voluntarily.

Ai2's Olmo 3 ships the decontamination tooling and the training data. NVIDIA's Nemotron 3 Ultra card names three harnesses, pinned containers and a reproducibility tutorial. Cohere's Command A report gives shot counts and aggregates over 10 runs, none of which appears on the model card most people open.

The most interesting disclosure came from OpenAI. The GPT-5.5 page footnotes its SWE-Bench Pro row with "Labs have noted evidence of memorization on this eval" and links to Anthropic's Claude Opus 4.7 announcement, which says "Our memorization screens flag a subset of problems in these SWE-bench evals." Two rival labs publicly agreeing a headline benchmark is partly memorized beats any leaderboard.

Is the contamination story real, or just repeated commentary?

I went looking for primary sources behind the famous claims, because this site has been burned by stats that trace back to nothing.

They mostly hold up. Scale AI's GSM1k paper commissioned a fresh clone of GSM8k and found "accuracy drops of up to 8%, with several families of models showing evidence of systematic overfitting". The FrontierMath conflict-of-interest story is documented by the benchmark's own maker: Epoch AI states that "OpenAI commissioned Epoch AI to produce 300 advanced math problems" and "retains ownership of these questions and has access to the problems and solutions, with the exception of a holdout set." The Leaderboard Illusion counted 27 private Llama-4 variants tested before release, and estimated Google and OpenAI received 19.2% and 20.4% of all Arena data.

What does not hold up is the maximalist version. Bordt et al., ICML 2025, found "even 144 times of contamination can be forgotten if the training data is scaled beyond five times Chinchilla." The GSM1k authors landed similarly: "many models, especially those on the frontier, show minimal signs of overfitting." Contamination is real, measurable, and smaller than the discourse implies.

So are AI benchmarks broken?

No. Benchmarks are instruments, and the instruments are fine. What is broken is the reporting layer on top of them.

Saturation is visible from inside the launch pages. OpenAI's GPT-5.2 page reports 100.0% on AIME 2025. A benchmark at 100% has stopped measuring anything, and the vendor printing that number is the one telling you so.

The distinction that matters: "benchmarks are broken" is a claim about measurement, and it is mostly false. "Benchmark marketing is unfalsifiable" is a claim about disclosure, and my count says it holds 23 times out of 23 at the item level and 11 out of 23 at the configuration level. A score with no harness, no setup and no outputs is not a measurement you can dispute. It is an assertion in a table.

What would change my mind?

One vendor publishing the per-task outputs behind one headline number. Not the dataset, which several already release. The model's actual answers, so anyone can regrade them. Until then, treat a launch table like any expert-level claim with no working shown: probably directionally right, entirely uncheckable, priced accordingly.