Free generator · SEO and Visibility
Robots.txt Generator.
Preselects the paths worth blocking on WordPress, Shopify or Next.js, offers the AI crawler groups with the trade-off spelled out, and tests any URL against the file as you build it.
Sets the paths worth blocking on that stack.
One per line, starting with a slash.
One per line, absolute. This line is read by every crawler, whether or not you have verified the site with them.
Blocking these keeps your content out of model training and, for some, out of AI answers. That second part cuts both ways — a citation in an AI answer is a referral you no longer get.
Your robots.txt
# robots.txt
# Generated at godesigntech.com/tools
User-agent: *
Disallow: /search
Disallow: /*?*sort=Save this as robots.txt in the site root, so it answers at /robots.txt. It only applies to the exact host and scheme it is served from — a separate subdomain needs its own file.
Googlebot can crawl /
No rule in the * group matches this path, so it is crawlable.
Check done
Want someone to fix what this found?
Finding a problem and fixing it are different jobs, and the second one is ours. Send the result over and you get back what it would take to put right — scoped and priced, from the person who would do the work, within one business day.
What this tool measures
Build a robots.txt for your platform with the paths worth blocking already selected, then test URLs against it before you upload anything.
The generator half is trivial and every site has one. The part worth having is the tester underneath. A rule that reads correctly and blocks the wrong thing is the characteristic failure of this file, and it does not announce itself — it shows up weeks later as pages quietly leaving the index, by which time nobody connects it to a one-line change.
How to use it
Pick your platform first. The preselected paths are the ones that are genuinely worth excluding on that stack — admin areas, cart and checkout, internal search, and the faceted filter parameters that multiply a store's URL count into the hundreds of thousands.
Add the sitemap URL. It is one line, it is read by every crawler including ones you have never verified with, and it is the cheapest discovery signal available.
Then test. Paste the paths of pages you expect to be crawlable and confirm they are, before the file goes live rather than after.
How to read the result
Blocking a path does not remove it from search. A blocked URL that other pages link to can still be listed, showing the bare URL with no description. robots.txt saves crawl budget; a noindex tag removes pages. Reaching for the wrong one is the most common mistake in this file.
The warning about blocking CSS and JavaScript is worth taking seriously. Googlebot renders pages before judging them, and a crawler denied the stylesheet renders yours broken and evaluates the mobile experience on that. Blocking /wp-content/ was standard advice in 2012 and has been actively harmful since 2015.
The AI crawler section presents a genuine trade-off rather than a recommendation. Blocking GPTBot and ClaudeBot keeps your content out of training data. Blocking OAI-SearchBot and PerplexityBot also keeps you out of AI answers, and a citation in an AI answer is a referral. Google-Extended is the cleanest case: it controls Gemini training only and has no effect on Google Search.
What to do next
Save the file as robots.txt in the site root so it answers at /robots.txt. It applies only to the exact host and scheme it is served from — a subdomain needs its own file, and so does a separate staging host.
Then check it live, and check it again after any platform migration. A new host serving a default robots.txt that disallows everything is one of the most common ways a relaunch loses its traffic in week one.
Questions people ask
Where does robots.txt go?
The root of the host, so it resolves at https://example.com/robots.txt. It cannot live in a subdirectory, and a file at example.com does not cover shop.example.com — subdomains are separate hosts and need their own.
Should I block AI crawlers?
It depends on what your content is for. A publisher whose business is the content itself has a real case for blocking training crawlers. A business whose content exists to be found has the opposite case, because being cited in an AI answer is increasingly how people arrive. The one thing not to do is block them without deciding.
What does Crawl-delay do?
It asks a crawler to wait between requests. Google ignores it entirely — crawl rate is managed in Search Console instead. Bing and Yandex honour it. It is worth setting only if a crawler is genuinely straining your server.
Can I use noindex in robots.txt?
No. Google supported it unofficially for years and stopped in 2019. Any line like that is silently ignored, which means pages people believe are hidden are crawlable and indexable. Use a meta robots tag or an X-Robots-Tag header instead.
Where to go from here
- Robots.txt Checker and URL Tester — Check the file once it is live, and test it as Googlebot would.
- XML Sitemap Checker — Validate the sitemap you just declared in it.
- SEO Audit Checklist — The rest of the crawl and index checks.
- The thinking behind the SEO audit checklist — How crawl control fits a full audit.
The guide behind this tool
Built by GoDesign FZE, who run technical SEO fixes for UAE sites.
Get in touch
Want someone to just do this part?
The tool is free and stays free. If you would rather hand the work over, tell us what you are dealing with and you get a straight answer on scope, cost and timeline.
- You own everythingRepository, hosting and domain credentials sit in your name from day one, not handed over at the end.
- A written timelineMilestone dates are agreed in writing before work starts, so you always know what ships next.
- One business dayEvery enquiry gets a reply from the person who would scope the work, not a sales sequence.
Media City, Dubai, UAE · DHA Phase 2, Islamabad, Pakistan
Would rather type than fill in a form?
Message on WhatsApp