I Checked 30 Paywalls. 22 Block Claude's Training Crawler. Only 13 Block the One That Reads on a User's Command.

TL;DR: Not on their own, and not in the way most publishers built them. A paywall stops a crawler that identifies itself and obeys robots.txt. It does not stop an agent browser driving a real Chrome session on a logged-in user's machine, which is what Columbia's Tow Center demonstrated in October 2025. On August 15, 2026 I fetched robots.txt and one representative article for 30 paywalled publishers. 22 of 27 resolved robots.txt files block ClaudeBot, the training crawler, but only 13 block Claude-User, the agent that fetches on a user's command, and only 9 of 18 article pages carry the complete paywall markup Google documents. The defense is concentrated on the layer that announces itself.

What did the Tow Center actually show?

Aisvarya Chandrasekar and Klaudia Jaźwińska published the test on October 30, 2025. They asked OpenAI's Atlas and Perplexity's Comet to retrieve the full text of a nine-thousand-word subscriber-exclusive article in MIT Technology Review. Both browsers did it. The same prompt in the standard ChatGPT and Perplexity interfaces failed, with both replying that they could not access the article because the Review had blocked the companies' crawlers.

The mechanism is the whole point, and it is not a hack. In the authors' words, "to a website, Atlas's AI agent is indistinguishable from a person using a standard Chrome browser." Every control publishers spent two years deploying operates on a request that voluntarily identifies itself. An agent driving a browser session does not send a crawler token, and it is often carrying the user's own cookies.

The second finding is the one publishers should sit with longer. Asked to summarize a New York Times article, Atlas did not break in. It avoided NYT content and built a summary from four other outlets: The Guardian, the Washington Post, Reuters and the Associated Press, three of which have licensing agreements with OpenAI. Blocking worked exactly as designed, and the answer got written from the competition. The Tow Center calls this a catch-22, and it means a successful block is not a win condition. It is a decision about who gets cited instead of you. The publishers trying to charge rather than refuse are not much further along: seven of 40 homepages returned 402 to an AI crawler and none named a price.

What I checked, and what I deliberately did not

I fetched two things per publisher, unauthenticated: robots.txt, and one representative article URL taken from the site's own homepage links. On the article I searched the served HTML for Google's paywalled content structured data: "isAccessibleForFree": false plus a hasPart block containing a cssSelector naming the gated element. In robots.txt I parsed user-agent groups and recorded four tokens separately: the training crawlers GPTBot and ClaudeBot, and the user-triggered agents ChatGPT-User and Claude-User, plus the search crawlers OAI-SearchBot and Claude-SearchBot.

I did not attempt to retrieve, reproduce or quote paywalled article text, and this post contains none. That is a design constraint, not an omission. The question here is what publishers are signalling and blocking, which is answerable entirely from robots.txt and public markup. Three robots.txt files refused me (thetimes.com, theinformation.com, newscientist.com) and 12 article fetches returned a challenge page or no parseable HTML, so the denominators are 27 and 18, not 30.

Which bots do paywalled publishers actually block?

The gap between the two layers is the finding:

Token	What it does	Disallowed
ClaudeBot	Anthropic training crawler	22 of 27
GPTBot	OpenAI training crawler	15 of 27
Claude-SearchBot	Anthropic search index	12 of 27
Claude-User	fetches on a user's request	13 of 27
ChatGPT-User	fetches on a user's request	12 of 27
OAI-SearchBot	OpenAI search index	9 of 27

ClaudeBot is blocked by 22 of 27. Claude-User, from the same vendor, by 13. Seventeen of 27 block at least one user-triggered agent, so the agent layer is not ignored, but 10 name none of them at all, and only 8 block both. The New York Times, The Economist, Bloomberg, telegraph.co.uk, chicagotribune.com, adage.com and globeandmail.com are the cleanest: blanket disallows across every token I tested. washingtonpost.com, ft.com, businessinsider.com, hbr.org, nationalgeographic.com and puck.news name none of the six.

Now read the vendor documentation against those numbers, because the two companies document opposite postures for the same category of fetch. OpenAI's crawler page says ChatGPT-User "is not used for crawling the web in an automatic fashion," and then, plainly: "Because these actions are initiated by a user, robots.txt rules may not apply." Anthropic's documentation says the opposite, that disabling Claude-User "prevents our system from retrieving your content in response to a user query," and that its bots respect robots.txt directives. One vendor tells you the line in your file may not bind. The other tells you it does. A publisher running one blocklist against both is getting two different products from the same three lines.

Does the markup tell anyone there is a paywall?

Partly. Of 18 article pages, 16 emit an isAccessibleForFree property at all. Twelve declare the sampled article paywalled with false. Only 9 carry the complete signal: isAccessibleForFree: false plus the hasPart block with a cssSelector identifying the gated element. Three declare the article paywalled without ever saying which part is gated. Two, businessinsider.com and sfchronicle.com, emit no isAccessibleForFree at all on the page I pulled.

One caveat that matters more than the headline. I sampled a representative article from each homepage, and four pages declared isAccessibleForFree: true. On a metered site, a free article is supposed to say that. So true is not necessarily an error, and my sample cannot distinguish "this publisher does not signal paywalls" from "this particular article is genuinely free." The 9 of 18 is a floor on correct implementation, not a ceiling on competence.

What does this not prove?

It does not prove leakage. This is the distinction the whole post rests on: a missing isAccessibleForFree is a signalling failure, not evidence that anyone read the article. Google's markup exists so a crawler can tell paywalled content from cloaking, which is a spam violation. Omitting it means Google is not being told the page is gated. It says nothing about whether any model, crawler or agent ever obtained the text, and I ran no test that could tell you that, by design.

Nor does robots.txt coverage measure enforcement. Every number in that table is a count of stated intent. Whether a given fetch honors it is a vendor promise in one case and an explicit maybe in the other, and in the agent-browser case documented by the Tow Center there is no token in the request to match against at all. The tokens are also a moving target, which is why a copy-paste blocklist misses most of what is actually crawling, and why the Comet browser's public security record is worth reading before assuming the agent layer behaves predictably.

So what actually holds?

Three things, in order of how much they are worth. Server-side authentication holds, because an unauthenticated request cannot render text it was never sent, and a cssSelector pointing at a div that is present in the HTML is a curtain, not a wall. Correct markup holds up your side of the bargain with Google, and 9 of 18 doing it fully is a bad number for an industry this loud about AI access. Robots.txt holds against the crawlers that agree to be held, which is genuinely most of the automated traffic and none of the agentic traffic. It also holds only if somebody wrote it, and across 73 customer sites on 29 hosted publishing platforms, 24 of 29 platforms name no AI crawler at all.

What none of it does is answer the Tow Center's real finding. When blocking works, the summary still gets written, sourced from whichever competitor left the door open. Deciding whether your content limits AI features is the same class of problem one layer up in Search, where publishers deployed a control scoped to a different product entirely. The pattern repeats: the control was chosen for the threat model of 2023, and the thing reading you now arrives looking like a person.