I Fetched 73 robots.txt Files Across 29 Hosted Platforms. Not One Platform Blocks the Bot That Cites You.

TL;DR: Mostly no. I fetched robots.txt from 73 live customer sites across 30 hosted platforms on August 15 and 16, 2026, and 29 platforms resolved. 24 of 29 name no AI crawler at all, so the platform makes no decision for you. The 5 that do decide (Medium, Squarespace, Tumblr, Write.as, and Substack when a writer flips a toggle) block only training crawlers. Zero platforms block OAI-SearchBot, ChatGPT-User, Claude-User or PerplexityBot, the agents that actually produce citations. The sites I found blocking search crawlers were all doing it by hand.

The premise I started with was that millions of writers publish on hosted platforms, never see a robots.txt, and get opted out of AI on day one by a host that never told them. It is a good premise. It is mostly wrong, and the way it is wrong is more interesting.

Which platforms actually name an AI crawler?

I picked 30 hosted publishing, store and site platforms, then fetched robots.txt from real customer sites on each rather than the platform's own marketing domain, because those files differ. Two to three sites per platform where I could, more for Substack. I confirmed each site was actually on the platform from response headers and page markup. 73 hostnames, 71 files retrieved, 2 confirmed absent.

Twenty-four of the 29 resolved platforms name none of the eight crawler tokens I looked for. That list is long and boring, which is the point: Ghost, beehiiv, Wix, Webflow, Framer, Notion Sites, WordPress.com, Blogger, dev.to, Hashnode, Bear Blog, Carrd, Gumroad, Shopify, BigCommerce, Kajabi, Teachable, Podia, Mailchimp Sites, Big Cartel, Buttondown, Micro.blog, Tilda, and Circle. Circle is the outlier inside the outlier: biblestudyco.circle.so and dddcommunity.circle.so both return 404 for robots.txt, so there is no file to read at all.

The five that do name something:

Platform	Sites sampled	AI crawlers blocked by default
Medium	medium.com, netflixtechblog.com, uxdesign.cc	GPTBot, ClaudeBot, Amazonbot, Applebot-Extended, Bytespider, FacebookBot, GoogleOther, meta-externalagent
Squarespace	pikeplacefish.com, drinkingwithchickens.com	26 agents including GPTBot, Google-Extended, ClaudeBot, CCBot, anthropic-ai
Tumblr	staff.tumblr.com, engineering.tumblr.com	CCBot, Google-Extended, ClaudeBot, anthropic-ai, Applebot-Extended, Amazonbot, and not GPTBot
Write.as	write.as	GPTBot, ClaudeBot, Amazonbot, Applebot, meta-externalagent
Substack	14 publications, see below	GPTBot, Google-Extended, on 6 of 14

Medium's file is byte-identical across its own domain and two custom-domain publications, so the block travels with you. Squarespace's is byte-identical across both sites I sampled and labels itself # Squarespace Robots Txt. Tumblr blocks Anthropic's training crawler and Google's but leaves OpenAI's alone, which I cannot explain and neither can the file.

Does Substack block GPTBot or not?

Both, and this is where a one-site sample would have produced a false headline. I fetched 14 distinct Substack publications. Six block GPTBot and Google-Extended and ship a Content-Signal line. Eight do not.

Blocking: newsletter.pragmaticengineer.com, honest-broker.com, natesilver.net, bariweiss.substack.com, heathercoxrichardson.substack.com, thebignewsletter.com. Not blocking: astralcodexten.substack.com, slowboring.com, garymarcus.substack.com, lennysnewsletter.com, experimental-history.com, derekthompson.org, chinatalk.media, erictopol.substack.com.

Every file shares the same Substack skeleton (Disallow: /lovestack/* gives it away), so this is one platform template with a per-publication switch, not a platform default. If I had checked Astral Codex Ten and published "Substack allows AI crawlers," I would have been wrong about six of fourteen publications. If I had checked The Pragmatic Engineer and published the opposite, wrong about eight. Anyone quoting a single sample for any of these platforms is quoting a coin flip.

The blocking publications also carry Content-Signal: search=yes, ai-input=yes, ai-train=no, and The Big Newsletter's file carries a second Cloudflare-managed block on top of Substack's, with the full Content-Signal preamble asserting Article 4 rights under the EU copyright directive. That is two independent layers of blocking on one newsletter, neither of which touches a search crawler.

Why does blocking GPTBot cost you nothing?

Because GPTBot is not how you get cited. OpenAI's crawler documentation is unambiguous: "GPTBot is used to make our generative AI foundation models more useful and safe," while "OAI-SearchBot is used to surface websites in search results in ChatGPT's search features." Two agents, two jobs. Anthropic's documentation splits three ways: ClaudeBot collects "web content that could potentially contribute to their training," Claude-SearchBot "navigates the web to improve search result quality," and Claude-User fetches pages "when individuals ask questions to Claude."

So a platform that blocks GPTBot and ClaudeBot has opted its writers out of model training and left every citation path open. That is the coherent posture, and all five of the platforms that block anything take it. Not one of the 71 platform-default files I read named OAI-SearchBot, ChatGPT-User, Claude-User or PerplexityBot. If you want the full token-by-token map of which crawler does what, I classified 30 of them by documented purpose here.

Who is blocking the search crawlers, then?

Individual site owners, by hand, on platforms that gave them a blank file. The clearest case is three BigCommerce merchants who each answered the same question differently on the same platform.

Mizuno USA and Chair King both serve the same 42-agent blocklist, and it includes OAI-SearchBot, ChatGPT-User and PerplexityBot alongside the training crawlers. Forty of those 42 agents appear verbatim in the ai.robots.txt community blocklist on GitHub, a 163-agent file that makes no distinction between training and retrieval. It is a copy-paste, and the copy-paste is what removes the store from ChatGPT's search results.

Saddleback Leather, on the same platform, did the opposite on purpose. Their robots.txt is commented like an internal memo: "Strategy: Allow AI answer engines, block AI training crawlers," with OAI-SearchBot, ChatGPT-User, PerplexityBot and ClaudeBot explicitly allowed and GPTBot and CCBot disallowed, under the header "OpenAI training crawler (NOT the search bot)." Three merchants, three postures, zero input from BigCommerce.

What does this not prove?

It does not prove your platform is not blocking you, because robots.txt is not the only lever. A platform can rate-limit, serve a challenge page, or render content client-side in a way that a crawler cannot read, and none of that appears in a text file. My check measures stated policy, not enforced behavior. It also does not tell you whether these crawlers obey the file at all; it only tells you what they were asked to do.

It also does not mean these defaults are permanent or that you are stuck with them. Wix's file ends with # Auto generated, go to SEO Tools > Robots.txt Editor to change this. beehiiv's says # beehiiv default robots.txt / This is automatically used when you leave custom content empty / Customize below or upload your own robots.txt file. Ghost gave 404 Media enough control to add a GPTBot block that platformer.news, aftermath.site and blog.pragmaticengineer.com do not have. These platforms are handing you the pen and telling you where it is.

And these files change without announcement. Everything above is what those 73 hosts served on August 15 and 16, 2026.

How do you check your own site in thirty seconds?

Open https://yourdomain.com/robots.txt in a browser and search the page for OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-User. If none of those appear, or they appear with Allow: /, the assistants can reach you and your absence from their answers is an authority and crawlability problem, not a permissions one. If any of them sits under a Disallow: /, you found your answer, and on 24 of these 29 platforms you can edit the file yourself in the SEO settings.

If your platform is one of the five, decide whether you actually mind. Training blocks cost you nothing in citations and they are the only thing any of these platforms turned on. The expensive mistake is the other one, and every instance of it I found was self-inflicted, usually by someone pasting a list they never read. That list is still doing more damage than llms.txt ever fixed.