Why Google Won't Index Your Page: A Checklist

Two problems that look identical and are not

Almost every "my page is not indexed" complaint is one of two things, and they need opposite responses.

A code fault is something wrong with the URL or the page. It is fixable today, and once it is fixed the page becomes eligible. A trust gate is Google having fetched the page and decided not to spend index space on it yet. There is nothing to fix, because nothing is broken — and this is where most advice sends people in circles.

Order matters because the two produce the same symptom in Search Console, and the fix for one does nothing for the other. Re-requesting indexing on a page in a trust gate wastes your one lever. Rewriting the content of a page blocked by a noindex tag is worse than wasted, because the actual problem is still sitting there.

Step 1 — read the status code, not the page

A page that renders in your browser is not proof of anything. Ask the server instead:

curl -sS -o /dev/null -w "%{http_code}" https://example.com/page

What the answer tells you:

• 404 or 410 — the page is not indexed by design. Either the URL is wrong or it was deliberately removed. Nothing else in this guide applies.
• 5xx — the server is failing. Google backs off and retries later; it will not index a page it cannot fetch. Fix the server first.
• 301 or 302 — you are looking at a different URL from the one you think. Follow the redirect and diagnose the destination.
• 200 — good, but keep reading, because 200 is where the real traps live.

The trap that hides inside a 200 is the soft 404: the server reports success and the content is an error page. Our robots.txt survey caught exactly this on two of 78 well-known domains — khanacademy.org and cdc.gov answered 200 with an HTML error page instead of a robots file. A monitor watching status codes sees nothing wrong and a browser shows a page. If your unindexed URL returns 200 and reads like an error, you have your answer.

Step 2 — is it blocked?

Two mechanisms can block indexing, and they are not the same mechanism:

• robots.txt — a Disallow line stops the crawl. Note that it stops the fetch, not the indexing of a URL Google already knows about from links elsewhere.
• A noindex meta tag or X-Robots-Tag header — this stops indexing even when the page is crawled normally.

One combination costs people weeks: a robots.txt Disallow and a noindex on the same page. The disallow prevents Google from fetching the page, so Google never sees the noindex, so the URL can still be indexed from links pointing at it. You then remove the noindex, observe no change, and conclude the tag was never the problem. The order to undo it is: remove the Disallow first, let Google fetch the page, confirm it reads the noindex, and only then remove that.

You can test the first of these with our robots.txt tester. For the header, one curl is enough: curl -sSI https://example.com/page | grep -i x-robots-tag. And note that a page with no robots meta at all is normal — index, follow is the default, so its absence is not a fault.

Step 3 — can it be discovered?

Google cannot index a URL it never finds. Three ways a page stays invisible:

• It is not in your sitemap. Check that the URL appears in sitemap.xml — and that the URL listed there actually exists. A sitemap full of 404s is a signal of neglect rather than of coverage.
• The sitemap is not declared. Listing it in robots.txt lets a crawler find it without being told twice.
• Nothing links to it. This is the most common cause of a page that is technically perfect and never gets indexed. A page with no internal links is an orphan, and orphans depend entirely on the sitemap being read.

That last one is the same class of omission our canonical survey found on 11 of 55 famous homepages: the tag was simply absent. Nobody forgot on purpose. Omission is the default state of a site nobody audits.

Step 4 — is it a duplicate?

If the page is crawlable, discoverable and unblocked, the next question is whether Google treats it as a second copy of something else. The canonical tag is the instruction that decides this, and the mistake that matters is comparing it against the wrong URL.

In our canonical survey, 43 of 55 homepages redirected at least once before serving content. If you compare your canonical against the URL you typed rather than the URL you were finally served, you will conclude you have a cross-domain canonical problem when you do not. Compare against the final URL after redirects. Then check that the canonical target itself returns 200 — two of the 43 we tested pointed at URLs that did not, and nothing warns you when that happens.

Three more duplicates worth ruling out: the http and https versions of the page, the www and non-www versions, and any tracking parameters that create a second address for the same content.

Step 5 — is the content actually there?

If the page is a single-page app, the HTML Google receives may contain almost nothing. Curl the URL and read the raw source rather than the rendered page. If the body is an empty container and a bundle of script tags, that is what a crawler may be evaluating.

This is not hypothetical. Two of the homepages in our SERP snippet survey returned client-rendered shells with no title in the served HTML at all. Google can render JavaScript, but it does so on a second pass, and a page that needs rendering is a page that is slower and less certain to be evaluated.

Step 6 — now it is a trust gate, and there is nothing to fix

If every check above passed, the page is eligible and Google is choosing not to index it yet. Search Console describes this in two ways:

• Discovered — currently not indexed: Google knows the URL exists and has not fetched it yet.
• Crawled — currently not indexed: Google fetched it and judged it not worth index space at this point.

Neither is a defect, and neither is a bug you can patch. The usual reasons are thin content, no internal links, a brand-new domain with no authority, or content close enough to pages that already exist that Google does not need another copy of it.

What actually moves it: content clearly more useful than what is already indexed, internal links from pages that are themselves indexed, genuine external links or mentions, and time. What does not move it: requesting indexing repeatedly, resubmitting the sitemap repeatedly, or rewriting the page for the third time.

We went through this on our own site. A batch of pages sat in "Crawled — currently not indexed" while every technical check passed: 200 status, self-referencing canonical, index and follow, well over a thousand words each. There was nothing to fix. They drained on their own within days. The useful conclusion is not "wait and hope" — it is that when all five steps above pass, the correct action is to stop working on that page and go make the rest of the site better.

What "Request indexing" does, and does not do

It queues a fetch. That is the whole of it. It is worth using on a brand-new URL, or on a page you have just fixed, because it shortens the wait for the next crawl.

It is not worth using on a page that has already been crawled and judged. Requesting indexing does not override a quality decision, there is a practical limit on how often you can use it, and spending it on a trust gate burns the one lever you have for a genuine fault.

The order, in one table

The point of working in this order is that you stop at the first failure, and you stop entirely at step 6. Most people start at step 6 and work backwards, which is why the advice to "just keep improving the content" gets applied to pages that were never crawlable in the first place.