22 Articles Quote Anthropic's Crawl-to-Refer Ratio. I Counted 22 Different Numbers.

TL;DR: No. I collected 26 articles published in 2026 that quote a crawl-to-refer ratio and 22 of them qualified, between them attributing 22 distinct values to Anthropic alone, from 1,683:1 to 286,930:1. Only 3 of the 22 carry the caveat Cloudflare published in the same paragraph as the metric, that referrals arriving without a referrer header make every ratio look worse than it is, and only 2 link the Cloudflare post that defines the number at all. The ratio is a real measurement of HTML requests against HTML referrals on Cloudflare's network in a specific week. It is not a measure of how much any AI company is taking from you.

"Anthropic crawls N pages for every visitor it sends" is the most-repeated publisher statistic of the year, and N in the wild spans a factor of 170.

What does Cloudflare's number actually count?

The metric comes from one post: The crawl before the fall… of referrals, by David Belson and Sam Rhea, published 1 July 2025. The definition is a single sentence, and it is narrower than almost every article that quotes it:

the total number of requests from relevant user agents associated with a given search or AI platform where the response was of Content-type: text/html, divided by the total number of requests for HTML content where the Referer header contained a hostname associated with a given search or AI platform.

Three things follow. It counts HTML only, not images, PDFs, JSON or anything else a crawler pulls. The denominator is not "visitors," it is HTML requests carrying a referrer header from that platform's hostnames. And it covers traffic on Cloudflare's network, so a site behind a different CDN contributes nothing to it.

The famous figure in that post is Anthropic at 70,900:1 for the week of 19 to 26 June 2025. The figure nobody repeats is the other end of the same table: Mistral at 0.1:1, sending ten times as many referrals as crawl requests. The original announcement of the metric contained an AI operator that was net-positive for publishers. That fact did not survive the first round of quoting.

Cloudflare also flagged its own biggest weakness in the same paragraph. Referral traffic from Claude's native app carries no Referer header, and Cloudflare believes the same holds for other native apps, so "these calculations may overstate the respective ratios, but it is unclear by how much." The number is known by its publisher to be too high by an unknown amount.

How many different numbers are in circulation?

I collected articles published in 2026 that quote a crawl-to-refer ratio for any operator, fetched each one, and recorded what it claimed. Twenty-six screened, 22 qualified (two never quoted a ratio, one was published in December 2025, one used the phrase without a per-operator figure).

Those 22 articles attribute 22 distinct values to Anthropic or ClaudeBot: 286,930, 70,900, 50,000, 43,214, 38,744, 38,065, 38,000, 23,951, 20,583, 13,528, 11,122, 10,300, 8,800, 5,143, 4,580, 4,039, 3,386, 2,947, 2,363, 2,237, 1,917 and 1,683, all to one.

That range is not the story, because the metric genuinely moved: Anthropic shipped web search, so the ratio fell. The story is what happens when you group by the window each article names. For June 2026, nobori.ai reports 4,580:1, websearchapi.ai reports 4,039:1, technologychecker.io reports 3,386:1, and websearchapi's monthly crawler report reports 2,947:1. For July 2026: 2,237:1, 1,917:1 and 1,683:1 from three different outlets. For Q1 2026: 23,951:1 and 20,583:1.

One window reconciles perfectly. Five separate articles report 11,122:1 for the week of 25 May to 1 June 2026, and they agree to the digit. It is also the most precisely specified window in the set. Precision about the window is the variable that predicts whether two articles agree, which tells you the disagreement is not about Anthropic's behaviour at all. It is about people reading a rolling dashboard on different days and calling the result "current."

Do the articles say which week they mean?

Better than I expected, and worse than it looks. Nineteen of 22 attach some window to their headline figure. But only 9 specify a week. The rest offer a month, a quarter or just "2026," and those are precisely the ones that fail to agree with each other.

The three that attach no window at all are the ones whose numbers will travel furthest, because a figure with no date on it never expires. Coronium calls 20,583:1 "the highest documented crawl-to-referral ratio of any major AI company" with no period given. guptadeepak.com publishes a clean table putting Anthropic at 50,000:1 across all sites and 8,800:1 in computer and electronics, undated. Something Inc prints a six-operator table, ClaudeBot at 5,143:1, with no window column. All three are the kind of citable, confident artefact that gets pasted into a slide in January and quoted in October.

The best behaved in the set is Kaliber, which publishes a "reference window" column next to every ratio, and Digital Applied, which prints the warning explicitly: ratios "are tied to a specific time window and must never be averaged across windows."

Where does the citation chain terminate?

Thirteen of the 22 link some cloudflare.com URL. Only 2 of 22 link the post that defines the metric. The rest cite the Radar homepage, an unrelated Cloudflare blog post, another secondary article, or nothing.

The consequence shows up in how the denominator gets described. Ten of 22 mention a referrer at all, and 3 of 22 say that the metric counts only HTML responses. Three of 22 apply the referrer-stripping caveat to the ratio itself, and exactly one, Digital Applied, attributes it to Cloudflare and names the cause: "native-app traffic, the Claude and ChatGPT mobile and desktop apps, often arrives without a Referer header, which can make these ratios overstate the imbalance." One article in 22 carried through the sentence the publisher wrote next to the number.

I could not read today's live value, and I am not going to invent one. Cloudflare Radar's AI Insights page returns a JavaScript challenge to a plain fetch, and the Radar API replies {"success":false,"errors":[{"code":9106,"message":"Missing X-Auth-Key, X-Auth-Email or Authorization headers"}]} without credentials. Every "current" number in this post is somebody else's reading of that dashboard on a day they mostly did not record.

What does this not prove?

It does not prove the ratio is wrong. Cloudflare's measurement is careful and its caveats are published; the failure is downstream, in a citation chain that drops the window and the definition within one hop.

It also does not prove any AI company is extracting more or less than the number suggests, because the ratio is not a harm metric. It measures HTML requests against HTML referrals on Cloudflare's network. Sites elsewhere are invisible to it. Referrals arriving without a Referer header, which by Cloudflare's own account includes Claude's native app, are counted as zero, so the true ratio is lower than the published one by an unknown margin. This is the same undercount that makes AI referral traffic in GA4 so easy to lie to yourself about.

And a falling ratio is not a crawler behaving better. Anthropic's improvement came from the denominator: Claude started sending referrals. The crawling did not stop. If you are using this number to decide whether to block a bot, that is a separate question with separate evidence, and I answered it by reading 24 robots.txt files instead. The token most publishers reach for first does not do what they think either: Google's documentation never scopes Google-Extended to AI Overviews.

Quote it if you like. Quote the week with it, or you are quoting a dashboard screenshot from a day you cannot name. That is the same failure I found across the AI-visibility trackers, where only five of 25 will tell you how their number is produced.