Home › Tools › robots.txt Tester

robots.txt Tester

Paste a robots.txt and test any URL against any user-agent. See exactly which rule wins, which line matched, and whether Googlebot can crawl the page.

Paste your robots.txt on the left, then enter a URL and a user-agent. The tester tells you allow or disallow, and — more usefully — quotes the exact rule that decided it, so you can find the line that is quietly blocking a section of your site.

How to use it

  1. Paste the full contents of your robots.txt (fetch it from /robots.txt first).
  2. Enter the path you want to check and pick a user-agent.
  3. Read the verdict — it names the winning rule and the line number it came from.

Longest match wins, not first match

This is the single most misread part of the spec. Crawlers compare every matching Allow and Disallow rule and follow the longest one. So Disallow: /admin/ plus Allow: /admin/public/ means /admin/public/ is crawlable, even though the disallow line comes first.

If you have ever 'fixed' a block by moving a line up or down in the file, this is why it did not work. Order inside a group is irrelevant; path length is what decides.

robots.txt blocks crawling, not indexing

A disallowed URL can still appear in search results if something else links to it — Google will show it with a 'page not crawled' note because it never read the content. If you need a page gone from the index, use noindex, and do not disallow it at the same time, or the crawler can never see the noindex.

This pairing — disallow plus noindex — is one of the most common ways sites accidentally keep dead pages in the index indefinitely.

What this tool deliberately does not do

It does not fetch the robots.txt from your domain for you. Browsers block reading a file from another origin unless that origin explicitly allows it, and routing the request through a proxy would mean this site could see every URL you test — exactly what a privacy-respecting tool should not be able to do.

It also does not know about rules that live outside robots.txt: noindex in a meta tag, an X-Robots-Tag response header, or password protection. A URL can be fully allowed by robots.txt and still never appear in search for any of those reasons.

Treat the verdict as 'what your robots.txt says', which is one layer of the picture, not the whole one.

Questions

Does Disallow match by prefix?

Yes. A rule of /blog also blocks /blogging and /blog/anything. Add a trailing slash when you mean the directory only, and remember that wildcards * and $ are supported by Google.

What if no rule matches?

The URL is allowed. Absence of a matching Disallow is an implicit allow — you do not need Allow: / to permit crawling.

Do all crawlers obey robots.txt?

Major search engines do. Many AI training crawlers and scrapers do not. robots.txt is a request, not an access control mechanism — anything that must stay private needs real authentication.

Related tools

Meta Tag Generator

Generate clean page title, meta description, canonical, Open Graph and Twitter card tags.

Heading Structure Analyzer

Paste HTML and get a clean H1–H6 outline plus a list of real problems: missing or duplicated H1, skipped levels, and headings that are too long.

Keyword Density Checker

Count single words, two-word and three-word phrases with stopwords filtered out.