URL Normalization in the Wild

How this was measured

On 2026-10-28 we made 13 HTTP requests against each of the same 78 hand-picked domains our other surveys use — news, commerce, SaaS, developer tooling, social, streaming, reference, SEO, education, finance, cloud, local and government — and on 2026-10-30 we added the three dot-segment probes back for every domain. All 1,014 requests were made with curl as a subprocess, direct connection first and a local proxy second; the split was 742 direct and 272 proxied. Every response body was stored, so every number below can be recomputed offline without touching the network.

The question is narrow, and it is one our canonical survey cannot answer. That survey read what each homepage declares about itself — whether it ships a <link rel="canonical"> and what that tag points at. This one asks what each server does when the same resource is requested under a different spelling: a trailing slash, a capital letter, a doubled slash, a dot segment, an extra tracking parameter. A canonical is a declaration; a redirect is a behaviour. A site can declare one perfectly and still serve six live URLs for the same file, and that is what we went looking for.

Four definitions decide every number, and each of them had to be chosen carefully.

• The reference is a known real file, not the homepage. We used /robots.txt: it exists on essentially every site, it is plain text, and it is small enough to fingerprint exactly. Any variant of the homepage would have been confounded by the homepage's own redirects. The homepage was measured separately, against itself.
• Duplicate means byte-identical visible text, not merely the same status code. A 200 tells you the server was willing to answer; it says nothing about whether you got the document you asked for. We strip scripts, styles and comments, collapse whitespace, and take a SHA-1 of what is left. Two responses count as the same document only if those fingerprints match exactly — and for the text file we also lowercase it, so /robots.txt and /ROBOTS.TXT are compared as text rather than as bytes.
• Moved means the path or the query changed, not the host. A bare domain that 301s to its www has not answered the URL-normalisation question; it has answered the host question, which is a different survey. We compare pathname + search only, so bbc.com to www.bbc.com counts as the same URL on a different host, while /robots.txt to /robots.txt/ counts as a genuine move.
• Every probe was asked twice. Two of the thirteen are controls: the homepage fetched a second time, and the homepage with a tracking parameter fetched a second time. Without them you cannot tell “the parameter produced a different document” from “this page changes on every request”, and the second explanation is common enough to matter. Finding 7 shows what the controls changed.

Sixteen synthetic assertions lock the parser before it touches real data (--selftest): a page whose only difference is an inline script must fingerprint identically, a page with a different title must not, locOf must drop host and scheme but keep the query, hostOf and pathOf must separate host from path, and the parameter-presence test must accept ?utm_source=x and reject a bare path.

One trap is worth stating up front, because it invalidated our first run. curl collapses /./ and /../ before it sends the request, so a probe for /./robots.txt never reaches the server in that form — the server receives /robots.txt and answers it perfectly. Our first version reported 68 of 68 sites as handling dot segments flawlessly, which was an artefact of the client, not a fact about the web. The fix is --path-as-is, and the three dot-segment probes here were re-fetched with it on 2026-10-30.

Finding 1: nine in ten sites serve a duplicate of at least one common URL variant

We asked each site for eight variants of a URL it already owns: the homepage with a tracking parameter, the homepage with a doubled leading slash, /index.html, and five ways of spelling the reference file (/robots.txt/, /ROBOTS.TXT, /./robots.txt, /probe-dir/../robots.txt, //robots.txt). Of the 78 domains, 69 had at least three of those probes give a usable answer, and those 69 are the sample for this finding.

62 of those 69 (89.9%) serve a byte-identical duplicate of at least one variant — the same document, reachable at a second URL, with no redirect standing between them. The distribution is not a gradient: 7 sites (10.1%) duplicate nothing, 8 duplicate exactly one variant, 14 duplicate two, 14 duplicate three, 14 duplicate four, 8 duplicate five, and 4 duplicate six.

The worst offenders are the ones a technical SEO audience would expect: github.com (6 of 8), kubernetes.io (6 of 7), pinterest.com (6 of 8) and twitch.tv (6 of 8), then figma.com, go.dev, netlify.com, notion.so, searchenginejournal.com and slack.com at 5 of 8 each. Only one site duplicated every variant it answered — searchengineland.com — and its margin was thin: three of its probes were refused with 403, so it answered on exactly three variants and duplicated all three.

The seven that duplicate nothing are ebay.com, etsy.com, linkedin.com, harvard.edu, coursera.org, booking.com and doordash.com. They are strict, but not all for the same reason: several simply 404 the wrong spelling instead of redirecting it, and a 404 is only a correct answer if the 404 page itself is a real 404 — which Finding 9 shows is not automatic.

Finding 2: the trailing slash is the most common leak

Of the 66 sites that gave a usable answer for /robots.txt/, 30 answered 404 (45.5%), 17 redirected (25.8%), 12 served the identical file (18.2%), 5 served something else entirely (7.6%), and one each hit a redirect loop and a 4xx. A trailing slash, it turns out, is usually treated as a genuinely different path rather than a spelling variant — which is the right behaviour, and it is also why a trailing slash is such an easy thing to get wrong when a site switches frameworks.

Of the 17 that redirected, 16 normalised the slash away — theguardian.com, shopify.com, slack.com, figma.com, docker.com, go.dev, youtube.com, screamingfrog.co.uk, yoast.com, searchenginejournal.com, harvard.edu, stanford.edu, stripe.com, coinbase.com, vercel.com and digitalocean.com all land on the slashless form. The seventeenth is spotify.com, which is not a normalisation at all: every request we made to it — including a path that does not exist — landed on https://www.spotify.com/int/why-not-available/. That is a regional block, and it is a caution for the whole survey rather than a finding about slashes.

The 12 that served the identical file split into two kinds, and the distinction matters more than the count. Six served it with no redirect whatsoever — github.com, gitlab.com, asana.com, kubernetes.io, nodejs.org and netlify.com answered /robots.txt/ with the exact bytes of /robots.txt, from the same URL, with zero hops. The other six (notion.so, reddit.com, pinterest.com, tumblr.com, paypal.com, airbnb.com) reached the same content after a host change. A redirect at least consolidates the two URLs onto one; no redirect leaves both live at once.

Finding 3: case sensitivity is the norm, not the exception

Ask for /ROBOTS.TXT and the verdict is unambiguous: 45 of 66 sites (68.2%) answer 404, 6 redirect (9.1%), 9 serve the identical file (13.6%), 4 serve something else, and 2 return a 4xx. Paths are case-sensitive, and servers enforce it. Only nine sites in this sample treat an uppercase path as the same resource.

Those nine are worth naming because they are the exception that breaks tools: walmart.com, notion.so, x.com, kubernetes.io, twitch.tv, stripe.com, netlify.com, cdc.gov and who.int. Of them, three served the uppercase URL with no redirect at all — x.com, kubernetes.io and stripe.com — so both spellings are live addresses for the same file.

The practical consequence is small for a text file and larger for pages: a site whose router is case-insensitive will happily serve /About and /about as two URLs. Case-sensitive routers throw the problem away for you, and on a case-insensitive one you have to catch it yourself.

Finding 4: dot segments are the one thing nearly everybody gets right

/./robots.txt and /probe-dir/../robots.txt are resolved by the web server before routing ever happens, exactly as RFC 3986 describes. The result shows: 46 of 65 sites (70.8%) serve the identical file for the /./ form, and 45 of 64 (70.3%) for the /../ form. The rest answer 404 (21.5% and 17.2%) or serve something else. Not one site in this sample redirected a dot-segment path.

This is the good news of the survey, and it is worth understanding why it is good news. Dot segments are the classic path-traversal and duplicate-URL vector, and here they collapse into the canonical path automatically — there is no second URL, no redirect to follow, nothing for a crawler to trip over. The 14 and 11 sites that 404 instead are being stricter than required, which is also fine: a 404 for a path that does not exist is a valid answer.

The catch is on the client side, and it is the reason this finding needed a re-fetch. If you test dot segments with an ordinary curl or with a browser, the client normalises the path before the request leaves your machine, so you are testing the canonical path and concluding everything is perfect. That is how our first run reported 68 out of 68. You need --path-as-is to send the path as written.

Finding 5: a doubled leading slash is a hole in 19 sites

//robots.txt is the variant with the largest number of live duplicates after the trailing slash. Of 68 usable answers, 32 redirected (47.1%) and 27 served the identical file (39.7%); only 7 answered 404. And of those 27, 19 served it with no redirect at all, including github.com, slack.com, kubernetes.io, nodejs.org, go.dev, developer.mozilla.org, stripe.com and netlify.com.

The homepage behaves differently, and the difference is diagnostic. Asking for // produced a redirect from 30 of the 58 sites that answered, but 11 served the identical homepage with no redirect. That contrast is the signature of a CDN or edge router normalising the path: the edge sees //, strips it and forwards the canonical path, so the origin never knows the difference. Sites that answer the homepage correctly while failing the file path are usually getting one from a rewrite rule at the application layer and the other from the server.

This is the variant most likely to appear in a crawler's URL list without anyone noticing, because it is not a link anyone publishes — it is what you get from a badly written concatenation of a base URL and an absolute path.

Finding 6: /index.html still resolves on a quarter of the sample

Of 59 usable answers for /index.html, 31 answered 404 (52.5%), 14 redirected (23.7%), 7 served the identical homepage (11.9%) and 3 served something else. The interesting detail is how the redirects and the duplicates split: two sites — kubernetes.io and rust-lang.org — serve /index.html with no redirect at all, and the other six duplicates arrive there after a host change.

There is a second way /index.html becomes a live URL that no redirect handles: cdc.gov canonicalises its homepage to /index.html. We were redirected from / to https://www.cdc.gov/index.html, and /index.html answers 200 with the homepage content. Any internal link that hardcodes /index.html lands on the same page as /, and the two are not interchangeable to a crawler.

Finding 7: a tracking parameter does not create a second document

This is the finding that needed the most care, and the control probes are what made it publishable. We asked for / and then for /?utm_source=serpprism-probe. Naively, 41 of 59 sites (69.5%) returned byte-identical content and 4 returned something different. But “something different” proves nothing on its own: a homepage with a rotating hero image changes between any two requests, parameter or not.

So we fetched the same URL twice and compared. Of 60 sites, only 44 (73.3%) returned byte-identical content on two consecutive requests to their own homepage; 16 were unstable, including cnn.com, target.com, asana.com and pinterest.com. Among the 43 stable sites, 38 (88.4%) returned byte-identical content for the parameterised URL, and the three that did not — w3.org, harvard.edu and coursera.org — also returned different content on their second parameterised request. Their pages change on every load; the parameter is not the cause.

The decisive number is therefore a zero. We cross-checked four fingerprints — homepage, homepage again, parameterised URL, parameterised URL again — and asked: is there any site where the two parameterised fetches match each other but differ from the homepage? There is not one. Across all 78 domains, no site produced a separate document for a tracking parameter.

This is the practical good news in the survey, and it is often stated badly. The claim that adding utm_source to a URL creates a new indexable page is wrong for these sites: the parameter rides along on the same document. The UTM link builder on this site exists to make the parameters consistent, not to make them safe — the safety comes from the server returning one document, and from the canonical tag pointing at one URL. We measured the first; the second is covered in what 55 homepages actually do with canonical.

Finding 8: when a redirect does happen, it usually carries your parameter with it

The interesting split is not how many sites redirect but what they change. Of the 55 sites whose parameterised request ended on a 200, 13 did not redirect at all, 33 redirected to a different host with the path and query untouched, and 9 changed the path or the query.

The 33 host changes are the bare-domain-to-www kind: bbc.com to www.bbc.com, notion.so to www.notion.com, zoom.us to zoom.com, mozilla.org to www.mozilla.org and so on. All 33 carried the tracking parameter through untouched. That is worth knowing precisely because it is the opposite of what people assume: a canonicalising redirect does not clean your URL for you. The tagged URL stays a real, reachable, distinct address, and it is still yours to consolidate with a canonical tag.

The 9 that changed the path split on the parameter itself. Five kept it: nodejs.org to /en?utm_source=serpprism-probe, developer.mozilla.org to /en-US/?utm_source=serpprism-probe, stripe.com to /cn?utm_source=serpprism-probe, paypal.com to /c2/webapps/mpp/home-general?utm_source=serpprism-probe and squareup.com to /us/en?utm_source=serpprism-probe. Four dropped it: theguardian.com to /international, linkedin.com to www.linkedin.cn/, wikipedia.org to / and cdc.gov to /index.html.

Five of those nine are locale redirects, not canonicalisation: stripe.com to /cn, linkedin.com to .cn, theguardian.com to /international, paypal.com to /c2/webapps/, squareup.com to /us/en. Our exit IP is in Asia, so we were served the Asian variants. A crawler geo-dispatched to a different region sees a different canonicalisation behaviour, which is a real operational risk and a real limit on this survey at the same time.

Finding 9: three sites answer 200 for a path that does not exist

We requested /serpprism-nonexistent-827193 on all 78 domains. 58 of the 65 that answered (89.2%) returned a proper 404, which is the correct answer and the good news of this finding. Three did not, and they fail in two different ways.

reddit.com and twitch.tv returned byte-identical content to their own homepage. Their fingerprints match the homepage fingerprint exactly. These are single-page applications: the server has no route for the path, so it serves the application shell and lets the client decide. The shell is the same shell, so the answer is a 200 carrying the homepage, and a crawler that treats 200 as “exists” has just been told a page that does not exist is a page that does.

trello.com returned a 200 carrying an error shell whose visible text reads “Your browser was unable to load all of Trello's resources. They may have been blocked by your firewall, proxy or browser configuration.” It is not the homepage and it does not say “not found”, but a crawler has no way to know that either.

One more site belongs in this section even though the classifier called it a redirect. pinterest.com moved our nonexistent path to /serpprism-nonexistent-827193/ (it added a trailing slash) and answered that with 200 and the application shell. The URL-level rule counts a move as a redirect, but the substance is the same as Reddit's: the path does not exist and the server says 200.

Two consequences follow. A crawler will eventually hit these paths, and a site that answers 200 for everything needs a real 404 handler, not a catch-all route. And for you auditing your own site, a 404 response is the only thing that proves a path does not exist — the status code, not the page content, not the title, not the fact that the page says it could not find what you asked for. The robots.txt tester on this site reads the rules themselves rather than guessing at intent, and our first robots.txt survey covers the same file from the directive side.

What to ship

Decide one URL per resource, then redirect everything else to it with a single hop. A site that 301s /robots.txt/ to /robots.txt is doing this correctly; a site that serves both is leaving two URLs alive. In this sample, 6 sites serve the identical file at both trailing-slash forms and 3 at both uppercase forms, with no redirect at all.

Point a canonical at the one URL, and make it return 200. The canonical tag is the only mechanism that survives a URL the server does not redirect — and we measured exactly how often that situation arises.

Do not rely on the trailing slash being normalised for you. 45.5% of this sample answers 404 for /robots.txt/, so a framework migration that changes the trailing-slash rule can quietly create a second URL for every page you have.

Expect a doubled slash to reach your application. 19 sites serve the identical file for //robots.txt with no redirect, and 11 serve the identical homepage for //. If your router normalises, that is fine; if it does not, you have duplicate URLs that no link ever creates.

Use a 404 for paths that do not exist. A catch-all route that returns 200 for every path is the single most expensive mistake in this survey, because it makes the absence of a page unobservable to a crawler.

Do not expect your server to strip tracking parameters. 38 of the 42 sites that redirected on a parameterised URL passed the parameter through to the destination. Your analytics parameters survive canonicalisation; only an explicit rule removes them, and a canonical tag is the right place to consolidate rather than to strip.

Test the path as written. If you check /./robots.txt with an ordinary HTTP client, you will test the canonical path and conclude everything is fine. Our first run did exactly that and reported 68 of 68 sites as perfect.

Our redirect chain survey measures the other half of this problem — the http to https to www hops that get a homepage to its final URL — and the sitemap survey measures which URLs a site asks to be indexed. Those are complements of this one: a canonical chain and a sitemap are both declarations, and this survey is about the behaviour that declarations are supposed to override.

Limitations

The reference file is a text file. /robots.txt is a static file served by the web server, so these results describe how the server handles path variants. A page routed through an application framework can behave differently — and usually does, because frameworks own their own routing table. The homepage probes partially cover that case, and the two disagree in exactly the way you would expect.

Ten probes is not a URL space. We tested eight variants plus two controls. A real crawl finds more: session identifiers, index.php versus index.html, mixed-case directory names, percent-encoded characters, doubled slashes in the middle of a path, and /./ after a query string. Every one of those is a separate experiment.

One snapshot, from an exit IP in Asia, over two network paths. Regional variants serve different markup and different routing rules, which is exactly what produced the geo-redirects in Finding 8 and the regional block on spotify.com. A crawler geo-dispatched elsewhere would see different behaviour from the same site, and we could not measure that.

18 domains never gave a usable answer — 403 from a WAF, a bot challenge, a connection failure — and they are not a random loss. Large consumer and financial sites that fight crawlers are over-represented among them, which biases the sample towards sites that are permissive about URL variants.

Content fingerprints are a proxy for identity. We compare visible text, which ignores differences in attributes, in script payloads and in byte-level formatting. Two responses with different fingerprints are certainly different documents; two with identical fingerprints are very probably the same document, but two documents could in principle differ only in a script. For the questions asked here that distinction does not change any conclusion, and for the reference file we additionally compare lowercased text.

The snapshot was taken across two days. The base run is 2026-10-28 and the dot-segment probes were re-fetched on 2026-10-30. A site that changed its routing between those two dates would be recorded inconsistently; we found no such case, but we did not look for one.

Reproduce it

The script and the shared domain list ship with this site. node scripts/survey-url-normalization.mjs fetches all 13 probes for every domain and writes .workbuddy/tmp/url-normalization-survey/raw.json; --report re-derives every number in this article from that file without touching the network, and --selftest asserts the parser against sixteen synthetic cases first. --evidence prints the original response behind each verdict — --evidence=query, =dslash, =slash, =case, =dot, =soft and =challenge narrow it to one class of finding, and every line carries its domain. Because the report is derived entirely from the stored responses, the three dot-segment probes can be re-fetched on their own with --only=refDot,refUp,refDS and merged into the existing snapshot, which is how they were corrected after the client-side path-collapsing bug.