Home › Tools › Robots.txt Generator
Robots.txt Generator
Build a robots.txt file rule by rule: choose crawlers, list the paths to block or allow, add your sitemap, then copy or download the file.
Crawlers the rules apply to:
SerpPrism's Robots.txt Generator is a free online tool that builds a robots.txt file from your choices: which crawlers the rules apply to, which paths to block, which to allow anyway, and where your sitemap lives. There is no sign-up and the file is assembled in your browser. It writes the file; it does not test one. If you already have a robots.txt and want to know whether a specific URL is allowed for Googlebot, that is the robots.txt Tester's job. The two tools are deliberately separate, because writing rules and predicting how a crawler applies them are different tasks. Build the file here, then paste it into the tester with a few URLs you care about before it goes live.
How to use it
- List the paths crawlers should stay out of, one per line — and any exceptions under 'allow anyway'.
- Pick which crawlers the rules bind. AI training crawlers get their own groups, so blocking them does not touch search indexing.
- Copy or download the file, place it at yoursite.com/robots.txt, then run three or four real URLs through the tester before you call it done.
Generator or tester: which one do you need
The generator assembles syntax: one group per crawler, rules in the right shape, the sitemap line where crawlers expect it. The tester simulates judgement: given a file and a URL, it applies the longest-match rule and tells you allowed or blocked, and which line decided it. Writing a file and predicting its effect are different skills, and the failure mode of mixing them up is a file that looks right and blocks the wrong thing.
The workflow this site is built around is: generate the file here, test it there with the handful of URLs whose fate you actually care about, publish, then test the live file once more. Four steps, and the middle one is the one everyone skips.
AI crawlers get their own groups, on purpose
A crawler with its own group in robots.txt follows that group and ignores the * group entirely. That single fact explains most hand-edited robots.txt mistakes of the last two years: a GPTBot line added under the * group does nothing, and a block-all under * does not stop a named bot that has its own group elsewhere in the file.
It is also good news. Blocking GPTBot, ClaudeBot or CCBot stops training crawls without touching search indexing, because Googlebot and Bingbot never read those groups. Google-Extended is the subtle one: it controls whether Google may use your content for Gemini training and grounding, and blocking it does not remove you from Google Search. This generator repeats the shared rules into every group you select, which is verbose and exactly how the format has to work.
Mistakes a generator cannot save you from
robots.txt is a public file. Listing a path in it announces that the path exists, so never put anything in a Disallow line that you would not say out loud — real protection is authentication, not a crawler request. And a disallowed page can still be indexed if other sites link to it; blocking crawling is not the same as removing a page, which is what noindex is for.
Each host needs its own file, too: www and non-www, and every subdomain, are separate hosts with separate robots.txt files. Our guide to common robots.txt mistakes covers these failure modes, including the ones that produce no error message anywhere.
Questions
Where do I put the generated file?
At the root of the host it should govern: https://yoursite.com/robots.txt, served as plain text. Subdomains each need their own file, and a file in a subdirectory is simply never found.
What is the difference between this and the robots.txt Tester?
This tool writes a file from your choices. The tester reads a file — yours or anyone's — and tells you whether a given URL is allowed for a given crawler, naming the rule that decided it. Generate here, verify there.
Does blocking an AI crawler affect my Google ranking?
No. GPTBot, ClaudeBot, CCBot and Google-Extended are read by AI training systems, not by the crawlers that build the search index. Blocking them changes who may train on your content; Googlebot and Bingbot are unaffected.
Can I block one folder for Googlebot but allow it for everyone else?
Yes — give Googlebot its own group containing that block. Remember that a crawler with its own group ignores the * group, so repeat any shared rules inside the Googlebot group too. The generator does this repetition for you when you tick both boxes.
Should I add a crawl-delay?
Only if a crawler is genuinely overloading a small server. Google ignores the directive entirely; Bing honours it. A large crawl-delay in the * group slows down every well-behaved crawler, including ones you want, so treat it as a last resort rather than a default.
Is an empty Disallow line valid?
Yes — 'Disallow:' with nothing after it means 'nothing is blocked', and it is how a group says allow-everything explicitly. The generator emits it for any selected crawler when you have listed no paths to block.