1. Disallowing a page you also marked noindex
This is the most damaging one and it looks reasonable. You want a page out of the index, so you add noindex — and you also disallow it so crawlers stop wasting time. But a crawler that cannot fetch the page can never read the noindex. The URL stays in the index indefinitely, usually with a note saying it was not crawled.
The fix is counterintuitive: to remove a page using noindex, you must allow crawling. Let Google fetch it, see the directive, and drop it. Only then disallow, if you still want to.
2. Blocking CSS and JavaScript
An older habit, and still common in older configs. Google renders pages, and rendering requires your stylesheets and scripts. Block /assets/ or /static/ and Google sees a page with no layout, which it may judge as broken or as a poor mobile experience.
Check your file for broad disallows on asset directories. If they exist, remove them — there is no upside to blocking Google from seeing your own stylesheet.
3. Assuming order decides priority
It does not. Within a user-agent group, the longest matching path wins regardless of line order. So Disallow: /admin/ followed by Allow: /admin/public/ leaves /admin/public/ crawlable. Reordering those lines changes nothing, which is why people conclude the file is being ignored when it is behaving exactly as specified.
4. Wildcard patterns that over-match
Disallow: /*.pdf$ blocks PDFs — and if written slightly wrong, blocks far more. A misplaced * or a missing $ turns a narrow rule into a site-wide block. Because robots.txt produces no warnings, this can run for months before anyone notices the traffic drop.
Test every rule containing a wildcard against a few real URLs before deploying it. That is what the tester on this site is for.
5. Forgetting that staging is one typo away from public
A staging host protected by robots.txt alone is not protected. Any crawler that ignores the file — and there are many — will index it. Duplicate staging content competing with production is a slow, confusing problem to untangle later.
Put real authentication in front of staging. Keep the robots.txt disallow as well, but understand it is politeness, not a lock.
A five-minute check that catches all of these
Fetch your robots.txt and paste it into a tester. Then run five URLs through it: your homepage, a category page, a deep article, one asset from your stylesheet directory, and a URL you believe is blocked. If any of the five gives a verdict you did not expect, you have found the bug — and you have found it before Google did.
Do this again after any deploy that touches routing, and any time someone new edits the file. The failure mode here is silence: nothing breaks visibly, nothing appears in an error log, and traffic from one section of the site quietly stops arriving. A monthly five-minute check is the entire maintenance cost.