Home › Tools › robots.txt Tester

robots.txt Tester

Test any URL against any user-agent and see the exact rule that wins, quoted with its line number. Built from 70 real robots.txt files.

By Hongtao Ren · Updated 2026-09-22

robots.txt

URL to test User-agent

Paste your robots.txt on the left, then enter a URL and a user-agent. The tester tells you allow or disallow, and — more usefully — quotes the exact rule that decided it, so you can find the line that is quietly blocking a section of your site.

How to use it

Paste the full contents of your robots.txt (fetch it from /robots.txt first).
Enter the path you want to check and pick a user-agent.
Read the verdict — it names the winning rule and the line number it came from.

Longest match wins, not first match

This is the single most misread part of the spec. Crawlers compare every matching Allow and Disallow rule and follow the longest one. So Disallow: /admin/ plus Allow: /admin/public/ means /admin/public/ is crawlable, even though the disallow line comes first.

If you have ever 'fixed' a block by moving a line up or down in the file, this is why it did not work. Order inside a group is irrelevant; path length is what decides.

What the exception pattern looks like in 70 real files

We parsed the robots.txt of 78 well-known domains and got usable files from 70 of them (45 directly, 25 through a proxy; 2 returned HTML instead of a robots.txt and 6 were unreachable). Inside those 70 files there are 27,443 directive lines. The pattern above is not rare — it is how large sites are written.

356 rule pairs have an Allow longer than a Disallow on the same prefix, meaning the allow wins for the URLs it covers. 148 of those 356 are the inversion pattern: Disallow: / for everything, then one directory let back in. bbc.com does exactly this for GPTBot — Disallow: / followed by Allow: /storyworks, so the crawler may read that one path and nothing else. bloomberg.com does it for Google-Extended with /professional, reuters.com for * with /plus/, and quora.com with /about/. The other 208 are a subdirectory opened inside a blocked one, like gitlab.com blocking /users and allowing /users/sign_in ten lines later.

The file order proves the point on its own: in 47 of those 356 pairs, the Allow line appears before the Disallow it overrides. reuters.com writes Allow: /plus/ at line 129 and Disallow: / at line 130. If first-match-won were true, Reuters would be closed to every crawler. It is not. Anyone editing that file by reordering lines is changing nothing.

robots.txt blocks crawling, not indexing

A disallowed URL can still appear in search results if something else links to it — Google will show it with a 'page not crawled' note because it never read the content. If you need a page gone from the index, use noindex, and do not disallow it at the same time, or the crawler can never see the noindex.

This pairing — disallow plus noindex — is one of the most common ways sites accidentally keep dead pages in the index indefinitely. The full sequence, including what to check when a page stays in the index after you fixed it, is in our guide to why pages are not indexed.

How big a real robots.txt gets

The median file in our sample is 2,199 bytes and 70 lines — small enough to read end to end. The largest is figma.com at 392,044 bytes across 9,100 lines, and linkedin.com is 120,190 bytes over 4,863 lines. At that size nobody reads the file; they grep it, which is how contradictory rules survive for years.

Group counts follow the same curve: the median file has 4 user-agent groups and the maximum is 527, again pinterest.com. A quarter of the files — 18 of 70 — contain at least one group that names a crawler and then lists no rules at all, and 17 of 70 have a group shared by several crawlers, where consecutive User-agent lines apply to one block of rules. nytimes.com opens with User-agent: * and User-agent: Googlebot on consecutive lines, so every rule under them applies to both. The largest shared group in the sample covers 294 crawler tokens in one block.

None of this is wrong. It is what happens when a file outlives the people who wrote it, and it is the reason a tester that names the winning line is more useful than one that only says allow or disallow.

The lines that quietly do nothing

Nine of the 70 files, 14 lines in total, contain Disallow: with nothing after it. An empty value blocks nothing — it is the documented way to say 'allow everything'. harvard.edu and vercel.com both do it inside the * group, where it looks like a global block and behaves like a blank line. The rest sit under a single named crawler: theguardian.com and youtube.com under Mediapartners-Google, twitch.tv under Amazonbot, wikipedia.org under IsraBot, w3.org under W3C-checklink, uber.com under Googlebot-Image and doordash.com under Googlebot. Whether each of those was deliberate or a copy-paste, the effect is the same: the line does nothing.

The # character only starts a comment when it is the first thing on the line. Written after a rule, it is part of the path. Two files in the sample have a # inside a value — forbes.com with Disallow: /sites/universityofphoenix/#57934e047260 and coursera.org with Disallow: /business/wp-content/uploads/ . Both are almost certainly accidents, and both mean the rule targets a path that does not exist while the path the author had in mind stays open. If you want to annotate a rule, put the comment on its own line above it.

Wildcards are common and unevenly spread: 53 of 70 files use * or $ somewhere, but the counts are dominated by a handful of sites — figma.com alone has 9,048 rule lines containing one. A wildcard is not a stricter version of the same rule, so piling them up buys nothing; it makes the file harder to reason about and easier to get wrong.

Crawl-delay, Sitemap and the lines nobody agrees on

Eleven files set a Crawl-delay, 14 lines in total, every value a plain number. They run from 1 (github.com, x.com, tumblr.com, semrush.com) to 600 — ten minutes — at searchengineland.com, with searchenginejournal.com at 60 and moz.com at 10. Google's documentation does not list Crawl-delay among the directives it honours, so writing it to slow Googlebot down does nothing; Bing and Yandex do read it.

github.com is also the one file in the sample that spells the directive both ways, Crawl-delay: 1 on line 8 and crawl-delay: 1 on line 332. Field names are case-insensitive, so both work; the point is that a file can be internally inconsistent for years without anyone noticing, because nothing errors.

54 of 70 files declare at least one Sitemap:, 833 lines between them, and 26 of them declare more than one — booking.com alone lists 434. That matters because the robots.txt is the only place a crawler is told where the sitemaps are; assuming /sitemap.xml is wrong more often than not. Our robots.txt survey covers that in detail.

Finally, three files carry directives that no crawler is obliged to understand: four use Cloudflare's Content-Signal (cloudflare.com, vercel.com, netlify.com, heroku.com), theguardian.com has a License: line and yoast.com a Schemamap: line. Harmless, and a reminder that robots.txt is an open-ended key-value file — anything unrecognised is ignored silently rather than rejected.

One crawler at a time

A crawler obeys exactly one group: the most specific user-agent group that matches it, never a combination. If a Googlebot group exists, Googlebot ignores the * group entirely, including the allows in it. This is the most common source of 'but I allowed it' — the allow is in *, the block is in the Googlebot group, and only one of the two is ever read.

Eight files in the sample write Disallow: / inside the * group: reuters.com, reddit.com, x.com, linkedin.com, pinterest.com, quora.com, netflix.com and yelp.com. Three of those then open a single path back up with a longer Allow, so the practical result is a whitelist rather than a shutdown. Reading the group you care about, on its own, is the whole job — which is why this tool makes you pick a user-agent instead of testing 'the file'.

The same one-group rule is what makes AI crawler policy hard to read: adding a GPTBot group does not merge with *, it replaces it for that one crawler. We measured how 70 files treat 22 AI crawler tokens in our AI crawler survey.

What to test before you believe the verdict

Four URLs are enough. Test the homepage — you want it allowed, and a stray Disallow: / in the wrong group shows up here immediately. Test a listing or category page, because those are the templates that end up blocked by accident. Test one URL you believe is blocked, to confirm the rule actually matches the way you wrote it. And test one URL you know should be blocked, like /admin/, because a rule that fails to match is invisible until something private appears in the index.

Then repeat the set once per crawler you care about: Googlebot, Googlebot-Image and bingbot at minimum, and any AI crawler token you have written a rule for. The answers differ per group, and a file that is correct for one can be wrong for another. The common mistakes — including the ones that produce no error at all — are catalogued in our guide to robots.txt mistakes.

What this tool deliberately does not do

It does not fetch the robots.txt from your domain for you. Browsers block reading a file from another origin unless that origin explicitly allows it, and routing the request through a proxy would mean this site could see every URL you test — exactly what a privacy-respecting tool should not be able to do.

It also does not know about rules that live outside robots.txt: noindex in a meta tag, an X-Robots-Tag response header, or password protection. A URL can be fully allowed by robots.txt and still never appear in search for any of those reasons.

Treat the verdict as 'what your robots.txt says', which is one layer of the picture, not the whole one.

Questions

Does Disallow match by prefix?

Yes. A rule of /blog also blocks /blogging and /blog/anything. Add a trailing slash when you mean the directory only, and remember that wildcards * and $ are supported by Google.

What if no rule matches?

The URL is allowed. Absence of a matching Disallow is an implicit allow — you do not need Allow: / to permit crawling.

Do all crawlers obey robots.txt?

Major search engines do. Many AI training crawlers and scrapers do not. robots.txt is a request, not an access control mechanism — anything that must stay private needs real authentication.

I have a Googlebot group and a * group. Which one applies?

Only the Googlebot group, in full. A crawler follows the single most specific matching user-agent group and ignores every other group in the file — including the Allow lines in the * group. If a rule seems to be ignored, check which group it lives in before you change the rule itself.

Can I use wildcards?

Google supports * for any sequence of characters and $ to anchor the end of a path, so Disallow: /*.pdf$ blocks PDFs anywhere on the site. Other crawlers vary, and a wildcard is not a stricter version of the plain rule — Disallow: /admin* and Disallow: /admin behave the same for most paths. In our sample, 53 of 70 files use a wildcard somewhere.

Does Crawl-delay slow Googlebot down?

No. Google's documentation does not list Crawl-delay as a directive it honours; Bing and Yandex do. In our sample 11 of 70 files set it, with values from 1 second to 600. If a crawler is hitting you too hard, the fix is rate limiting at the edge, not a line in this file.

How do I remove a page that is already indexed?

Use noindex — a meta tag or an X-Robots-Tag header — and leave the URL crawlable so the crawler can see it. A Disallow keeps the crawler out and therefore keeps the noindex unread, which is how pages stay in the index for months. Remove the Disallow, wait for the noindex to be seen, then decide whether to block it.

How often is the file re-read?

Usually daily, and the result is cached. A change you make now is not guaranteed to take effect on the next crawl, so a rule that appears to do nothing for a day is not evidence that the rule is wrong. Google's URL Inspection tool fetches the current file when you need a definitive answer.

Can I block one crawler without listing every other one?

Yes — write a group for that crawler and no * group rules will apply to it. The catch works both ways: the * group's Allow lines do not carry over either. In our sample bbc.com uses exactly this shape for GPTBot, with Disallow: / and a single Allow for /storyworks.