Why the failure is silent
A robots.txt problem does not produce an error page, a server alert or a broken layout. The page still renders, the deploy still succeeds, and Search Console may keep showing the URL while the crawl that would update it has quietly stopped. The reason these mistakes stay hidden for weeks is that the file is read once, the decision is remembered, and nothing tells you afterwards that the remembered decision was wrong.
That is why the checklist below is ordered by the mistake that is easiest to make, not by the mistake that is easiest to find. The first item is the one that costs the most time, and it is also the one that looks the most sensible on the day it is written.
1. Disallowing a page you also marked noindex
This is the most damaging one and it looks reasonable. You want a page out of the index, so you add noindex — and you also disallow it so crawlers stop wasting time. But a crawler that cannot fetch the page can never read the noindex. The URL stays in the index indefinitely, usually with a note saying it was not crawled.
The fix is counterintuitive: to remove a page using noindex, you must allow crawling. Let Google fetch it, see the directive, and drop it. Only then disallow, if you still want to. The ordering matters: a disallow placed before the noindex is ever read converts a one-step removal into a two-step process with a waiting period in the middle.
This is also the mistake that produces the most confusing Search Console output, because the page is known, it is not crawlable, and the two facts sit next to each other with no explanation. The robots.txt survey found that a plain-text check catches the class of problem that status-code monitors miss; the same applies here. The file parses cleanly, the URL exists, and the only thing wrong is the relationship between two directives that were written at different times by different assumptions.
2. Blocking CSS and JavaScript
An older habit, and still common in older configs. Google renders pages, and rendering requires your stylesheets and scripts. Block /assets/ or /static/ and Google sees a page with no layout, which it may judge as broken or as a poor mobile experience.
Check your file for broad disallows on asset directories. If they exist, remove them — there is no upside to blocking Google from seeing your own stylesheet. The same rule applies to build-output directories such as /build/ or /_next/static/, which are often disallowed because they look like generated junk and then turn out to be exactly the files the renderer needs.
If the page still looks right in your browser after you remove the asset disallows, that is expected: your browser is loading the assets through the same rules the page itself uses, while the crawler's evaluation path is separate. The fix is still the same, but the symptom may not be visible to the person who introduced the block in the first place.
3. Assuming order decides priority
It does not. Within a user-agent group, the longest matching path wins regardless of line order. So Disallow: /admin/ followed by Allow: /admin/public/ leaves /admin/public/ crawlable. Reordering those lines changes nothing, which is why people conclude the file is being ignored when it is behaving exactly as specified.
The version that trips people up is the inverse: a broad Allow: / somewhere in the file followed by a narrow Disallow: /api/. The narrow rule still wins for the API path, and the presence of the broad allow does not cancel it. The only thing that cancels a rule is a longer, more specific rule on the same user-agent group, and the specificity comparison happens per request, not per line in the file.
4. Wildcard patterns that over-match
Disallow: /*.pdf$ blocks PDFs — and if written slightly wrong, blocks far more. A misplaced * or a missing $ turns a narrow rule into a site-wide block. Because robots.txt produces no warnings, this can run for months before anyone notices the traffic drop.
The dangerous shape is a wildcard with no terminator: Disallow: /tag/* and Disallow: /tag are not equivalent, and the second one quietly matches everything below /tag on the same host. If the intention was to block a specific suffix, the rule needs the anchor and the exact pattern. If the intention was to block a whole section, the section rule is correct and the wildcard is the thing to remove. Test every rule containing a wildcard against at least three real URLs before deploying it: the URL you want blocked, one sibling you want allowed, and one nested path you want allowed. That is exactly what the robots.txt tester is for.
5. Forgetting that staging is one typo away from public
A staging host protected by robots.txt alone is not protected. Any crawler that ignores the file — and there are many — will index it. Duplicate staging content competing with production is a slow, confusing problem to untangle later, especially when the staging URL carries the same product descriptions and is reachable from a link you published by mistake.
Put real authentication in front of staging. Keep the robots.txt disallow as well, but understand it is politeness, not a lock. The same applies to anything that should not be public: a private API, a test checkout, an unreleased campaign page. A disallow line is a signal to well-behaved crawlers, not an access control mechanism, and the two jobs need different tools.
The symptom map: which mistake does your Search Console output point at?
Most of these mistakes show up in the same place, so the useful skill is not rereading the file but matching the report line to the cause. The mapping is not one-to-one, but it narrows the space fast enough that the next step is always the same: open the file and check the user-agent group that the report says is affected.
| Search Console says | Most likely cause | First thing to check |
|---|---|---|
| "Excluded" with a noindex tag, but you did not set one | A disallow is preventing the tag from being read, and the exclusion note is stale | Whether the URL is disallowed in robots.txt at all |
| "Crawled — currently not indexed" | A trust gate, not a robots.txt fault | Whether the page is thin or orphaned, per the indexing checklist |
| Page dropped from index after a deploy | A new broad disallow or an over-matched wildcard | The diff of robots.txt on that deploy |
| Assets or scripts are not being crawled | A disallow on the asset directory | Any broad rule that matches the build path |
| Two URLs compete for the same query | Usually not robots.txt at all | The canonical tag, per the canonical survey |
The last row matters because people reach for robots.txt on duplicate problems it does not solve. A robots.txt entry decides whether a URL is fetched; it does not decide which URL is the one to keep. If two versions of a page are both crawlable, the decision is made by the canonical tag or by redirecting one of them, and the meta tag generator emits the tag for that case.
A before/after snapshot that catches the regression
The maintenance habit that catches every one of the above is a two-line one, and it costs nothing: store the last known good file, and diff it after any deploy that touches routing, asset paths or the staging environment.
curl -sS https://example.com/robots.txt | shasum gives you a fingerprint you can keep in a deploy log. When traffic from a section of the site stops arriving, the question is not "did someone edit robots.txt?" — it is "did the fingerprint change on the deploy that coincides with the drop?" A diff of two versions of a file this small is readable in a minute, and it turns a months-long traffic mystery into a four-line change with a commit hash attached.
The same habit makes the five-minute check below safe to skip: if the fingerprint is unchanged and the symptoms are unchanged, you have not introduced a new robots.txt problem, and you should go look at the pre-publish checklist instead of the file.
A five-minute check that catches all of these
Fetch your robots.txt and paste it into a tester. Then run five URLs through it: your homepage, a category page, a deep article, one asset from your stylesheet directory, and a URL you believe is blocked. If any of the five gives a verdict you did not expect, you have found the bug — and you have found it before Google did.
Do this again after any deploy that touches routing, and any time someone new edits the file. The failure mode here is silence: nothing breaks visibly, nothing appears in an error log, and traffic from one section of the site quietly stops arriving. A monthly five-minute check is the entire maintenance cost.
For the parts of the story that involve discovery rather than blocking, the llms.txt survey and the llms.txt generator cover the other half of the question: how the page is found once it is allowed.