How this was measured
On 2026-09-23 we fetched https://<domain>/robots.txt for 78 well-known domains and parsed 70 usable files โ 45 over a direct connection and 25 over a local proxy. Eight produced nothing we could read: stackoverflow.com answered 418, npmjs.com and nih.gov returned 403, vimeo.com timed out, rust-lang.org and wikimedia.org returned 404, and khanacademy.org and cdc.gov answered 200 with an HTML error page instead of a robots file. All eight are counted as missing rather than silently dropped.
In each file we built the user-agent groups, then asked one question for each of 22 known AI crawler tokens: is this crawler allowed to fetch /? A crawler with no group of its own inherits the * group, which is how robots.txt has always worked. Three verdicts are possible: blocked (root disallowed), partial (some paths disallowed, root open), allowed. Path matching follows Google's rule โ the longest matching pattern wins, and an equal-length Allow beats a Disallow.
Every number below was checked against the raw lines before it was written down. Where a claim involves a specific site, the exact lines were printed and read. That step mattered: two earlier surveys in this series each produced a batch of findings that looked entirely reasonable in summary form and were wrong in the source.
The one caveat that bounds everything
robots.txt is a request, not a lock, and it is increasingly not where the real decision lives. A site hosted behind Cloudflare can block AI crawlers at the firewall with a single switch and never mention them in robots.txt. Sites can also pursue non-compliant crawlers through terms of service and rate limiting. So every figure here describes what the file says, not what the crawler experiences. A crawler that ignores robots.txt is invisible to this method in both directions.
Finding 1: nearly two-thirds never mention an AI crawler
44 of the 70 files contain no AI crawler token at all. Not a block, not an allow โ silence. The list includes wikipedia.org, mozilla.org, w3.org, archive.org, python.org, nodejs.org, go.dev, kubernetes.io, youtube.com, spotify.com, stripe.com, paypal.com, coinbase.com, vercel.com, digitalocean.com, heroku.com, mit.edu, harvard.edu, stanford.edu, etsy.com, shopify.com, walmart.com and target.com.
Silence is not neutral, and it is not the same thing as permission-by-indifference: it means the crawler falls under whatever the * group says. For 42 of those 44, that works out to allowed. For two it does not โ reddit.com and pinterest.com both ship User-agent: * / Disallow: /, so they block every AI crawler without naming a single one. Reddit's file is two lines long and its comment points at a public content policy rather than at crawler rules.
The SEO industry is mostly silent too. ahrefs.com, semrush.com, screamingfrog.co.uk, yoast.com and searchengineland.com name no AI crawler. Only two companies in that category say anything: searchenginejournal.com (blocks omgili and Omgilibot) and moz.com, discussed below. The vendors whose tools audit robots.txt files have largely not used them to make a statement about AI.
Finding 2: the sites that do speak overwhelmingly block
26 files name at least one AI crawler. Of those, 22 block at least one, and 13 block every AI crawler they name: theguardian.com, bbc.com, cnn.com, washingtonpost.com, bloomberg.com, amazon.com, notion.so, figma.com, x.com, tumblr.com, searchenginejournal.com, yelp.com and who.int.
Only four name AI crawlers and block none: cloudflare.com, netlify.com, moz.com and twitch.tv โ and one of those four blocks nothing because of a mistake rather than a policy, which we come back to below.
The shape of the result is that AI crawler policy is concentrated in publishing. News organisations and marketplaces have made a decision; developer tooling, reference sites and universities mostly have not.
Finding 3: the most famous crawler is blocked the least
This was the counterintuitive result. Among the files that name a crawler, the block rate runs opposite to name recognition:
GPTBot is named by 17 sites and blocked by 10 (59%), with 5 more applying partial rules. ClaudeBot is named by 18 and blocked by 13 (72%). CCBot is named by 15 and blocked by 13 (87%). And the long tail is unanimous: Bytespider is named by 11 and blocked by all 11, Applebot-Extended by 10 of 10, omgili by 9 of 9, Diffbot by 8 of 8.
Read that as a decision rule rather than a ranking. Sites that think about this at all block the crawlers that exist to harvest training data โ Common Crawl feeds a large share of public training sets, and ByteDance's crawler has no search product attached to it โ while giving the crawler attached to the most visible AI product the most nuanced treatment. GPTBot is also the one most likely to get a partial rule instead of a blanket block, which is what a site writes when it wants the crawler to see some sections and not others.
Finding 4: two large sites went the other way, loudly
cloudflare.com is the clearest statement in the sample. Its file opens with User-agent: * / Allow: /, then carries a comment โ "Allow AI crawlers to access markdown versions of pages" โ followed by explicit Allow: / lines for GPTBot, ChatGPT-User, Google-Extended, Anthropic-AI, Claude-Web, CCBot, PerplexityBot and cohere-ai. It is not an absence of blocking; it is an affirmative allow, written per crawler, by the company that sells most of the blocking.
netflix.com is the inverted case. Its * group is Disallow: / โ everyone is out โ but the allowlist group that follows begins User-agent: googlebot and runs through Applebot, bingbot, Baiduspider, Yandex, facebookexternalhit, GPTBot, ChatGPT-User, OAI-SearchBot and Google-Extended, opening with Allow: /. Netflix shuts the door on generic crawlers and holds it open for the AI ones. Whatever the reasoning, it is the opposite of the pattern the headlines describe.
Finding 5: when blocking is selective, it is oddly specific
Blanket blocks are the norm, so the selective ones stand out โ and they are selective in ways that reveal intent.
moz.com writes User-agent: GPTBot / Disallow: /blog/ / Disallow: /learn/seo/. An SEO company has blocked AI crawlers from exactly its blog and its SEO education library, and from nothing else.
ebay.com splits OpenAI's own crawlers against each other: GPTBot and Applebot-Extended share a Disallow: / group with a short list of exceptions, while OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User get only parameter-level rules such as Disallow: /*_kw. That is the distinction between a crawler that trains on your pages and one that fetches a page because a person asked a question โ and it is the distinction most files in this sample never make.
linkedin.com does something similar and less consistent: GPTBot and ChatGPT-User blocked, OAI-SearchBot limited to specific paths like /public-profile/.
github.com puts GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai and PerplexityBot in one shared group with Crawl-delay: 1 and an allowlist of marketing pages, then gives Bytespider its own Disallow: /. It is also the only site in the sample that sets a crawl delay specifically for AI crawlers โ worth knowing that Google ignores Crawl-delay entirely, as do most AI crawlers.
Finding 6: the lines that read like a block and are not
twitch.tv ships User-agent: Amazonbot followed by Disallow: with nothing after the colon. An empty Disallow means nothing is disallowed; it is the same as allowing everything. The file names a crawler, appears to have a rule for it, and has no rule at all. One site out of 70, and exactly the kind of line a human reviewer would read as deliberate.
Two other mechanical details worth knowing. who.int writes Disallow:/ with no space, which is legal and works. And theguardian.com follows its block with License: https://theguardian.com/license.xml โ not part of the robots.txt specification, read by no crawler, and clearly aimed at people rather than machines.
Grouping style varies and it matters for maintenance. theguardian.com, github.com, yelp.com and washingtonpost.com put many crawler names into one shared block; nytimes.com, bbc.com, bloomberg.com and amazon.com use one block per crawler. Both are valid, but in the shared-block style a single edit changes the fate of twenty crawlers at once โ which is how a rule for one bot silently becomes a rule for all of them.
What we did not find
Across 70 files we found zero misspelled AI crawler tokens. We looked for the plausible mistakes โ openai, chatgpt, claude, anthropic, gpt, gemini, ai, bot โ on the theory that a token no crawler matches is the most common way to write a rule that does nothing. None appeared. We also found no site blocking Googlebot apart from reddit.com, which blocks everyone.
Two non-findings in a row across this series now. That is not an argument for ignoring the warnings โ an unmatched token is still a bug when you write one โ but it is an argument for checking how often the thing you are about to spend a week defending against actually happens.
What to do with this
Decide per crawler, not for "AI". The files that do this well split training crawlers from search and user-triggered ones. Blocking GPTBot and accidentally blocking OAI-SearchBot with it removes you from a different surface than the one you were aiming at.
Check what an unnamed crawler inherits. If your * group is Disallow: /, you have already made the AI decision without writing it down โ which is where Reddit and Pinterest ended up.
Read your file's empty directives. Disallow: with no value allows everything. If you meant to block, it is not blocking.
Then test it. Paste the file into a robots.txt tester and run one AI crawler token against your homepage, an article and a URL you believe is blocked. The failure mode here is the same one described in our piece on robots.txt mistakes: nothing errors, nothing logs, and the effect only shows up months later as missing traffic or missing citations.
If you want the earlier baseline for the same file, our first robots.txt survey covers sitemap declaration across 78 domains.
Limitations
70 files from 78 hand-picked domains spanning news, commerce, SaaS, developer tooling, education and government โ representative of well-known sites, not of the web. A random sample would be dominated by small sites with different incentives and would almost certainly show a lower rate of explicit AI policy.
The tokens are 22 specific crawlers chosen because they are documented and widely discussed. New ones appear constantly and none of them are measured here. A site may have a policy for a crawler we did not look for.
The larger limitation is the one from the top: robots.txt is advisory, and for AI crawlers it is increasingly not where enforcement happens. Firewalls, rate limits and terms of service decide most of this, and none are visible in the file. Treat these numbers as a survey of stated intent.
Finally, a single-day snapshot of files that change without notice.
Reproduce it
The script and the shared domain list ship with this site, run in about two minutes, and store the raw file for every domain โ so every claim above can be checked against the source rather than against our summary of it. If a row is wrong, the file settles it; corrections via the contact page.