Hype Index: 'AI Now Matches Human Experts on Real Work'

TL;DR: GDPval-AA v2's top score is 1,849 Elo, anchored to a human baseline of 1,000. Elo has an arbitrary origin, so that is not "1.8x a human" and does not convert to a win percentage. I audited 18 articles quoting a GDPval "matches human experts" figure and 16 loaded: 7 disclose the score is Elo from pairwise comparison, 4 disclose that v2's judge is a model rather than a human, and 0 disclose that the open task set is 220 of 1,320. The benchmark is serious work. The sentence built on it is not.

What is the claim?

The load-bearing version, as it travels:

AI now matches or beats human experts on real economic work. The top model scores 1,849 against a human baseline of 1,000.

Two things are true here and one is a category error.

Why can't 1,849 be turned into a percentage?

Because Elo is not a score out of anything. Artificial Analysis describes its own method plainly: "For each matchup, the two outputs are anonymized and an LLM judge picks a winner. These blind pairwise results are aggregated into an Elo rating per model." The leaderboard is labelled "Anchored to a human baseline of 1,000."

Anchored is the operative word. The 1,000 is a label Artificial Analysis assigned to the human reference point, exactly as chess federations assigned theirs. An Elo scale has an arbitrary origin and an arbitrary unit: add 500 to every rating in the pool and not one ordering, not one implied matchup outcome, changes. A quantity whose zero point is a convention cannot be put in a ratio. So "1,849 versus 1,000" is not 1.8x, not 85% better, and not a percentage of tasks won.

What Elo does carry is differences, which map to implied preference probabilities through a logistic curve. That conversion needs the scale constant the rating system was fit with, and the Artificial Analysis page does not publish one. So I am not going to print a win probability, because I would be inventing the parameter that produces it. The honest statement is the ordinal one: on 220 tasks, an LLM judge preferred that model's deliverables to the human reference deliverables more often than not.

At the time I read the page, the top three were Claude Opus 5 (Adaptive Reasoning, Max Effort) at 1,849, Claude Opus 5 (Adaptive Reasoning, Xhigh Effort) at 1,817, and Grok 4.6 (high) at 1,746, with 27 of 202 models carrying a published result. Those version strings are copied off the page, not from memory.

Which GDPval are we talking about?

This is the trap almost nobody flags, and why my count exists. Two different evaluations share a name.

OpenAI's original GDPval, published 2025-09-25, is graded by people. OpenAI's description: expert graders "blindly compare model-generated deliverables with those produced by task writers (not knowing which is AI versus human generated)." Human professionals from the same occupations, seeing two unlabelled files.

Artificial Analysis's GDPval-AA v2 reuses OpenAI's 220 open tasks, runs models agentically with shell access and web browsing through a harness it calls Stirrup, and hands the comparison to an LLM judge. Same dataset lineage, different grader species, different output format.

The much-repeated "matches or beats experts about half the time" belongs to the 2025 human-graded run, not to v2. OpenAI's own wording is that Claude Opus 4.1 "produced outputs rated as good as or better than humans in just under half the tasks." That is wins plus ties, on the 220-task gold set, judged by people, in 2025. Attaching that sentence to a 2026 Elo leaderboard fuses two evaluations into a claim neither one makes.

OpenAI's page cannot keep its own trend straight either: the prose says performance "more than doubled" from GPT-4o to GPT-5, while the chart caption on the same page says it "more than tripled in a year."

What did the count show?

I took 18 articles quoting a GDPval "matches or beats human experts" figure and fetched each. Two returned HTTP 429 and never loaded (aiwiki.ai and morphllm.com), so the denominator is 16. For each I checked three disclosures.

Disclosure	Count
(a) Score is Elo derived from pairwise comparison, not a percentage of tasks	7 of 16
(b) GDPval-AA v2's judge is a model, not a human	4 of 16
(c) The open gold set is 220 of 1,320 tasks	0 of 16

Fourteen of the 16 use the word "Elo" somewhere. Only seven explain what it is derived from. Nine reference GDPval-AA or Artificial Analysis by name, meaning nine are quoting the LLM-judged leaderboard, but only four say a model did the judging.

The third row is the one that surprised me. Three articles mention 220. Three mention 1,320. Not one mentions both, so not one tells a reader that the public leaderboard runs on a sixth of the benchmark. Transformer discloses the LLM-judge issue and cites 220; Epoch AI cites 1,320. BenchLM explains the pairwise Elo derivation cleanly and never gives a task count.

How many of the 44 occupations actually produce a document?

GDPval covers 44 occupations across 9 industries: 1,320 tasks, 30 per occupation, with 5 per occupation in the open gold set, written by professionals averaging 14 years of experience and given an average of 5 rounds of review. That sourcing is more rigorous than most benchmarks manage.

I tabulated the published list. Three things fall out.

First, 9 industries at "the 5 occupations within each industry" should give 45. The published list has 44, because retail trade carries four, not five.

Second, 5 of the 44 titles literally begin "First-line supervisors of": police and detectives, production and operating workers, office and administrative support workers, retail sales workers, and non-retail sales workers. Those are jobs whose output is other people doing work.

Third, my own classification, flagged as a judgement rather than a measurement: sorting each occupation by whether its defining output is a file a model can emit or an action in the physical world, I get 28 document-shaped and 16 artifact-blocked. The 16 include nurses, nurse practitioners, pharmacists, concierges, recreation workers, shipping clerks, private detectives, audio and video technicians, film and video editors, producers and directors, and the five supervisor roles.

OpenAI concedes the mechanism itself. An occupation qualified for inclusion if "at least 60% of its component tasks were classified as not involving physical work or manual labor," which means up to 40% of a listed occupation's real work can be physical and it still made the list. You can watch the conversion happen in OpenAI's own example task: a manufacturing engineer is asked to design a jig in 3D modelling software, and the deliverable is narrowed to "only a pdf document summarizing the design, using snapshots of the 3d design created."

What does this not prove?

It does not prove the models are bad at this. Producing a deliverable an occupational expert prefers to a professional's own file, blind, is a real result, and OpenAI's 2025 human-graded run is the strongest evidence for it. Nor is the Elo leaderboard wrong: as a ranking of models against each other, it does exactly what it says.

What it cannot support is the leap from "preferred in pairwise comparison on 220 one-shot tasks" to "matches human experts on real work." OpenAI names the gap itself: the evaluation is one-shot, so it never tests the revision loop, and its 100x speed and cost figures "do not capture the human oversight, iteration, and integration steps required in real workplace settings." It is the same gap behind what ChatGPT's work mode can actually finish and most AI coding productivity studies, and the same pattern as the METR time-horizon chart losing its success threshold: a number with a stated scope reaches the reader with the scope filed off.

Verdict: 65% noise / 35% signal. GDPval is one of the better-built benchmarks in the field, and the human-graded 2025 result is worth taking seriously. The 1,849 is not a multiple of a human, the v2 judge is a model, and the leaderboard runs on 220 of 1,320 tasks. Quote the number if you like. Quote the three qualifiers with it.