Cloudflare Splits Search, Agent and Training on September 15. I Read 40 robots.txt Files: 5 Sites Make That Split Themselves.

TL;DR: Probably not, and the scope is narrower than the headlines suggest. Cloudflare's new defaults apply to new domains onboarding to Cloudflare, and on those, Training and Agent are blocked by default only on the pages that display ads, while Search stays allowed. Existing customers keep what they have unless they change it. The real risk is slower: the three-way Search/Agent/Training split turns blocking agents into a checkbox, and almost nobody has drawn that line themselves. I fetched robots.txt from 40 major retail, travel, news and SaaS domains on 2026-08-15. 38 resolved, 15 name an agent-class user agent at all, and only 5 give agents different rules from training crawlers.

Who is this for?

You run something that fetches web pages on a live person's behalf: a research agent, a shopping agent, a monitoring job somebody starts by hand. You are not a training crawler and not a search indexer. Until this summer the web had no vocabulary for that difference. On September 15 it becomes a setting.

What is actually changing on September 15?

Cloudflare's July 1, 2026 post sorts automated traffic into three buckets and defines each one plainly. Search is "any behavior that collects or indexes your content, so it can answer questions about it later." Agent is "automated behavior that is acting, usually in real time, on a person's behalf, to get something done right now." Training is "a crawler taking your content to train or fine-tune a model."

Then the date: "On September 15, 2026, we'll be setting new defaults for each of these three classifications." The scope clause does most of the work here. The defaults land on new domains onboarding to Cloudflare, and on those, "Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default." Ad-monetized pages. New domains. Existing customers are told to mark their preference in Security settings before the date.

September 15 does not darken the web. It ships the category.

What does the robots.txt record show?

On August 15, 2026 I fetched robots.txt from 40 named domains, ten each from retail, travel, news and SaaS. 38 resolved. I parsed each file into user-agent groups, merged duplicates per RFC 9309, then compared every agent-class token against its training counterpart: GPTBot against ChatGPT-User, ClaudeBot against Claude-User, PerplexityBot against Perplexity-User.

Category	Domains fetched	Names an agent UA	Splits agent from training
Retail	amazon, walmart, target, bestbuy*, homedepot, costco, ikea, etsy, wayfair, nike	1	0
Travel	booking, expedia, airbnb, tripadvisor, marriott, hilton, delta, united*, kayak, southwest	4	2
News	nytimes, washingtonpost, theguardian, bbc, cnn, reuters, wsj, bloomberg, forbes, npr	9	2
SaaS	salesforce, hubspot, shopify, stripe, atlassian, zoom, slack, notion, figma, asana	1	1

*bestbuy.com and united.com refused every fetcher I tried. Best Buy returned 403 Access Denied to a Googlebot string; United reset the connection. Neither is a robots.txt decision, which is the point of step 4.

5 of 38 sites currently distinguish an agent from a scraper. Figma, Forbes and Tripadvisor do it the way Cloudflare is about to: GPTBot and ClaudeBot get a flat Disallow: /, while ChatGPT-User, Claude-User and OAI-SearchBot get a real allowlist. Figma's is the sharpest: agents get the homepage and the pricing pages in fourteen languages, nothing else. Reuters and Kayak arrive differently, giving an agent token its own group without naming a training crawler at all.

The other 10 that name an agent token give it rules byte-identical to the scraper's. Bloomberg lists ChatGPT-User, GPTBot, ClaudeBot, Claude-User and OAI-SearchBot and hands all five the same five-path allowlist. Airbnb's ChatGPT-User block is a copy-paste of its GPTBot block. Amazon, CNN and the New York Times wall off every AI token at once, undifferentiated. And 23 of 38 name no agent token whatsoever, which means Cloudflare's default is the only opinion their infrastructure will ever express on the subject.

What are the steps?

1. Pick a stable product token and publish who you are. Put a real name and a contact URL in your user agent. Failure mode: borrowing a browser string. I set a Chrome user agent for this count and Airbnb bounced me to a JavaScript domain-switch page instead of the file; plain curl with no user agent got the real robots.txt. The disguise cost me the fetch. Cloudflare's bar for Verified status is that "a bot operator needs to show two things: that you represent yourself honestly, and you don't abuse the access that honesty earns."

2. Sign your requests, and expect to sign them twice. Cloudflare's Web Bot Auth wants an Ed25519 key pair, a JSON Web Key Set hosted at /.well-known/http-message-signatures-directory, and three headers on every request: Signature-Input carrying tag="web-bot-auth", Signature, and Signature-Agent pointing at your directory over HTTPS. Failure mode: assuming one key satisfies everyone. Shopify's changelog of May 7, 2026 says "bots and agents that don't sign their requests are subject to the strictest limits," effective May 30, and its signed tier wants HTTP Message Signatures per RFC 9421 using ECDSA P-256, verified against your UCP profile. Different curve, different directory, same week's work.

3. Fetch the robots.txt of your destinations now, and keep the file. You need a September 14 baseline to diff against, matched on the token you actually send. Failure mode: reading someone else's rules as yours. OpenAI's documentation is explicit that ChatGPT-User covers "certain user actions in ChatGPT and Custom GPTs." If you built your own agent, that line does not govern you, and the sites that allowlist ChatGPT-User have said nothing about your token.

4. Classify the failure before you react to it. A refusal carries its reason. Cloudflare returns 402 Payment Required with a crawler-price: USD 0.01 header when a page is monetized rather than closed, though none of the seven sites I found returning 402 in the wild actually sent that header, site owners on paid plans choose 403 or 402 as their block response, and since December 2025 error responses carry a crawler-error header with eleven codes. Failure mode: treating every 4xx as a ban. A 429 is a rate limit, a 402 is an invoice, and a geo-redirect is not a block at all.

5. Never retry disguised. Failure mode: believing it works. It mostly does not, because the edge fingerprints TLS and IP reputation rather than reading your string, and when it does work you have traded a recoverable block for a permanent one. Cloudflare says bots that abuse content signals lose Verified status, and that takes every site which allowlisted you with it.

What does this count not prove?

Forty hand-picked domains are not the web, and robots.txt is only one gate. My two failures, Best Buy and United, blocked me at the edge without a robots.txt rule ever being consulted, which is how most agent blocks will actually arrive. The count also cannot see a Cloudflare dashboard toggle, and after September 15 the toggle is where the decision lives. What the record establishes is that the distinction Cloudflare is about to make on 38 sites' behalf is one only 5 have made themselves. The rest will inherit an opinion, the same way thousands of sites inherited an AI block as a CDN default in 2025.

Watch one more signal: PACT, backed by Chrome, Edge, Firefox and Shopify, lets sites issue anonymous tokens attesting to "personhood." The web is building positive proof of a human, not just better walls, and a headless agent will not have one.

How do you know it is working?

Log five fields on every outbound fetch: host, the token you sent, HTTP status, the crawler-price and crawler-error headers when present, and whether the request was signed. Your metric is reachability rate per host, weekly, with a hard cut at September 15.

My baseline, run unsigned and unauthenticated on public robots.txt: 38 of 40 hosts reachable, 95%. That is the ceiling for an anonymous agent on the easiest file on the internet. If your number drops after the 15th, step 4 tells you whether you are blocked, throttled or billed, and only one of the three has a fix that costs nothing. Agents fail quietly, which is why every agent demo works until you give it real work and why the disclosures on browsing agents repay reading before you scale one.