How this was measured
This is the same 78 hand-picked domains our other surveys use — news, commerce, SaaS, developer tooling, social, streaming, reference, SEO, education, finance, cloud, local and government — measured from a different angle. The robots.txt survey asks whether a crawler may fetch; this one asks what the page says about being indexed, in the two places a page can say it: the HTML and the HTTP response header.
The two layers were measured on two different dates, and that matters. The HTML layer reuses the homepage snapshots our SERP snippet survey stored on 2026-09-26 — reusing rather than re-fetching is deliberate, because it makes the 59 homepages here the same 59 pages those articles describe. The response-header layer was fetched fresh on 2026-10-15, because X-Robots-Tag lives in the response and was never stored. 59 of 78 homepages were readable (75.6%) and 63 of 78 returned a 200 response (80.8%) to the header pass, attempted directly first and through a local proxy second; 51 domains answered on the direct path and 27 needed the proxy.
Four parsing rules decided most of what follows. Unquoted attributes are real: kubernetes.io writes <meta name=robots content="index, follow">, and a parser that insists on quotes reports that site as declaring nothing. Content is split on commas, semicolons and whitespace, because content="index,follow" and content="index, follow" are the same two directives. Only the last real response block counts when reading headers: with a proxy in the path, curl writes its own HTTP/1.1 200 Connection established before the site's response, and treating that as the answer invents a site. And the bot-scoped header syntax is split before counting — X-Robots-Tag: bingbot: noarchive is one directive aimed at one crawler, not a directive called bingbot:.
Finding 1: most homepages say nothing at all
34 of 59 (57.6%) carry no <meta name="robots"> at all. Of the 58 homepages where we have both layers — readable HTML and a 200 response — 32 (55.2%) say nothing in either place. Silence is not an oversight in most of those cases. Everything a crawler does by default is what index, follow asks for, so a page with no directive and a page with index, follow are treated identically.
Which is why the more interesting number is what happens among the sites that do speak.
Finding 2: half of the declarations are declarations of the default
25 of 59 (42.4%) declare a robots meta tag, and 12 of those 25 declare nothing but index, follow, all or archive — every one of which restates the default. walmart.com ships INDEX,FOLLOW; target.com, x.com, nodejs.org, wise.com and wikimedia.org ship index, follow; moz.com ships all and cdc.gov ships index, archive. A few do not even bother with follow: twitch.tv and vimeo.com declare index alone, and vercel.com declares index, max-image-preview:large.
Harmless, but not pointless to know about: a tag that already exists is the tag someone edits later. The staging rule a developer adds at 2am, the noindex a plugin toggles during a migration — both land in an existing tag far more often than they create a new one. 14 of 59 (23.7%) carry at least one directive that is not a plain restatement of the default, and that is the whole set worth reading.
Finding 3: the directives that do something are almost all about previews
Across 66 directive tokens, the counts are: index 19, follow 15, max-image-preview:large 11, max-snippet:-1 7, max-video-preview:-1 7, and then one each of noindex, nofollow, all, archive, noodp, noydir and nocache.
Only two of those change anything for Google: max-image-preview:large and noindex. max-snippet:-1 and max-video-preview:-1 mean "no limit", which is what the default already is — so seven sites ship a pair of directives whose net effect is zero. archive is the default too. noodp and noydir were dropped by Google years ago: bbc.com still emits content="NOODP, NOYDIR", and nothing reads it. nocache is a Bing directive, and the one site using it, tumblr.com, scopes it correctly: <meta data-rh="" content="nocache" name="bingbot">. It is also the only crawler-specific meta tag in all 59 homepages.
The 11 sites asking for large image previews are theguardian.com, forbes.com, bloomberg.com, gitlab.com, docker.com, screamingfrog.co.uk, yoast.com, searchenginejournal.com, harvard.edu, heroku.com and vercel.com. That is the one setting here with a visible payoff: it is what lets a thumbnail render large in mobile results and Discover, and the default is standard, not large.
Seven of them also set the two max- limits, and six of those seven are byte-identical, down to the order: index, follow, max-image-preview:large, max-snippet:-1, max-video-preview:-1 appears on gitlab.com, screamingfrog.co.uk, yoast.com, searchenginejournal.com, harvard.edu and heroku.com; docker.com declares the same five with the order changed. Six homepages from six unrelated organisations do not independently converge on the same string — one CMS default got copied, and three of the six belong to SEO companies, which is the kind of coincidence that tells you where the default came from.
Finding 4: one homepage noindexes itself, and we checked it twice
airbnb.com serves <meta name="robots" content="noindex, nofollow"> on its homepage. It is not a stub: the document we stored is 595 KB with a real <title> and real content. Because a noindex that drastic is exactly the kind of result our own tooling could manufacture, we re-requested it with a normal Chrome user agent over a direct connection and got the same tag back — so it is not a response aimed at survey bots.
We cannot tell you why, and we are not going to guess: it may be deliberate, it may be a template that leaks onto one route. What it demonstrates is the diagnostic order. A homepage that will not index is almost never a mystery — it is a directive somewhere, and the search is for the directive, not for a penalty.
Our first pass reported two such sites, and the second one is the reason every verdict here gets printed before it gets published. paypal.com also returned noindex,nofollow — on an 8 KB interstitial with no <title> whose entire visible text is "Security Challenge. Please wait while we perform security check". That page is not a homepage, its noindex is not a finding, and including it would have doubled the most quotable number in this article. It is excluded now, and the check that excludes it reads the page as text first: the phrase is interrupted by tags, so matching it against raw HTML misses it.
Finding 5: X-Robots-Tag is nearly unused, and half of what exists is aimed at one crawler
4 of the 63 homepages that returned 200 send an X-Robots-Tag header (6.3%): theguardian.com and bbc.com send bingbot: noarchive, netflix.com sends index and moz.com sends all. Two of the four use the bot-scoped form, which is the one thing a header can express that a single HTML tag cannot cleanly: a rule for one crawler, in a response that goes to all of them.
netflix.com is the only site in the sample whose only declaration is the header — its HTML carries no robots tag at all. That is the argument for reading headers: a directive can sit in a place your HTML audit never looks, applied to a page whose markup is clean. Three sites declare in both layers and the two never disagree (theguardian.com and bbc.com pair an HTML max-image-preview:large with a header rule for Bingbot; moz.com says all twice). 22 sites declare only in HTML.
Finding 6: data-nosnippet is used on navigation, not on content
21 elements across 5 of the 59 homepages carry data-nosnippet, which tells Google not to use that stretch of text in a snippet. The distribution is the story: 15 of the 21 belong to one site, heroku.com, and all 15 are the same element — <span class="mega-description-group" data-nosnippet>, the descriptions inside its mega-menu. The rest: two on shopify.com, two on airbnb.com, one on kubernetes.io and one on mozilla.org, the last being a consent banner.
So the element-level control is being used for what it is good at — keeping interface furniture out of a snippet that would otherwise quote it — and it is being used by a handful of sites, with one site accounting for most of the instances. If you are deciding whether to adopt it, that is the realistic scope: a targeted fix for boilerplate that sits next to real copy, not a page-wide setting.
What to ship
Nothing, if you have nothing to say. 32 of the 58 homepages we could read in both layers declare nothing anywhere, and they are not being penalised for it. Adding index, follow for comfort adds a tag that someone will later edit by mistake.
max-image-preview:large, if you care how the result looks. It is the one directive in this sample that changes a Google default in your favour, and 11 sites use it. Skip max-snippet:-1 and max-video-preview:-1 unless you are setting them to a real number — -1 means "no limit", which is where you already are.
Check the header as well as the HTML. One site here declares only in X-Robots-Tag, and a staging rule that leaks into an edge header is invisible to any check that only parses markup. curl -sSI https://yourdomain/ shows it in one line.
Retire noodp and noydir. They have not been supported for years, and one site still ships them — harmless, but it is a tag that says someone stopped auditing.
Use data-nosnippet on the furniture. Mega-menu descriptions, consent banners, cookie text: the things a snippet would quote if you let it.
To see what your own page emits before you change it, the meta tag generator writes the title, description, canonical and robots tag from one input so they cannot drift apart, and the robots.txt tester covers the file that governs fetching rather than indexing. The AI crawler survey measures the third layer — per-bot rules in robots.txt, where the interesting disagreements actually live.
Limitations
The two layers were measured 19 days apart. The HTML comes from a 2026-09-26 snapshot; the response headers were fetched on 2026-10-15. A site could have changed a directive in between, and the 15 domains that did not return 200 in the header pass (403 from ebay.com, stackoverflow.com, npmjs.com, quora.com, searchengineland.com, coinbase.com, yelp.com and nih.gov; 401 from reuters.com; 406 from uber.com; 202 from amazon.com and booking.com; 301 from washingtonpost.com; 403 from nytimes.com) are counted as undecidable, not as absent.
18 of 78 homepages were unreadable in the HTML layer — 17 that returned nothing usable and 2 interstitial pages (khanacademy.org, paypal.com). The same large, heavily defended sites drop out of the heading, favicon and readability surveys, so the sample is biased towards sites that are easy to crawl.
We read server HTML only. A robots tag written by JavaScript after load is invisible to this method. It is rare — and no site in the sample looked like it was doing it — but the method cannot rule it out.
We requested the HTML homepage and nothing else. X-Robots-Tag is most often applied to PDFs, images and other non-HTML responses, which this survey never asked for; 4 of 63 is the rate on homepages, not on the web. We followed redirects and read the final response's headers, so a directive attached to an early hop would not appear here.
We measured what our user agent was served. Requests identified themselves as a survey crawler, and WAFs answer differently per client. We re-checked airbnb.com with a browser user agent because its verdict was the load-bearing one; the other 58 were not re-checked, and some of what we call silence could be a bot-specific response.
Reproduce it
The script and the shared domain list ship with this site. node scripts/survey-indexing-directives.mjs fetches the response headers and derives every number above; --report recomputes all of it from stored data without touching the network; --evidence prints the original meta tag and the original header line behind every verdict, which is how the paypal.com interstitial and the bingbot: syntax were both caught. The HTML layer reads the same snapshot as scripts/survey-serp-snippet.mjs and scripts/survey-description.mjs, so the three articles describe the same 59 pages. Our robots.txt, canonical, hreflang and AI crawler surveys run the same domain list from other angles.