Free checker · SEO and Visibility
Robots.txt Checker and URL Tester.
Fetches a site's robots.txt, reports the mistakes that quietly cost indexation, and tests any URL against the file using Google's own matching rules: most specific rule wins, Allow beats Disallow on a tie.
Enter any page on the site. robots.txt only ever lives at the root, so that is where this looks.
Check done
Want someone to fix what this found?
Finding a problem and fixing it are different jobs, and the second one is ours. Send the result over and you get back what it would take to put right — scoped and priced, from the person who would do the work, within one business day.
What this tool measures
This fetches a site's robots.txt, reads it the way Googlebot does, and answers the question people actually arrive with: is this specific URL blocked.
That question is genuinely hard to answer by reading the file, which is why the tester matters more than the listing. Two rules make it counter-intuitive. The most specific rule wins regardless of the order it appears in, so a Disallow near the top can be overridden by an Allow near the bottom. And a group naming a crawler replaces the wildcard group for that crawler entirely, rather than adding to it — so a two-line Googlebot group means Googlebot ignores every rule you wrote under the asterisk.
How to use it
Enter any address on the site. robots.txt only exists at the root of a host, so the tool goes there regardless of the path you paste.
Then use the tester. Paste the path of a page you expect to be crawlable — or one you expect to be blocked — and pick the crawler. Googlebot and Bingbot behave the same way here; the AI crawlers are worth checking separately, because a site that blocks GPTBot and ClaudeBot has made a decision about AI answers that is often nobody's decision in particular.
How to read the result
The tester names the rule that decided the outcome and the line it is on, so you can go and change that specific line rather than rewriting the file.
The most serious finding this tool reports is a 5xx response. Google treats a server error on robots.txt as a temporary disallow-everything, and if it persists it starts dropping pages. A site that returns 500 on robots.txt for a fortnight can lose most of its index without a single page having changed.
The second most serious is a noindex directive inside robots.txt. Google stopped supporting it in 2019 and it is silently ignored — so every page someone thought was hidden has been crawlable and indexable since then.
What to do next
If a page you want indexed is blocked, remove the rule rather than adding an Allow above it. Layered rules are how these files become unreadable, and the file has to stay readable because the person editing it next will not be you.
If a page you want hidden is crawlable, robots.txt was the wrong instrument. Blocking a URL in robots.txt does not remove it from search — it can still appear, listed without a description, on the strength of links pointing at it. To keep something out of results, allow the crawl and serve a noindex tag on the page.
Questions people ask
Does blocking a page in robots.txt remove it from Google?
No. It stops the crawl, not the indexing. A blocked URL that other pages link to can still be listed, showing the URL with no title or description — which usually looks worse than the page would have. Use a noindex tag for removal and reserve robots.txt for saving crawl budget.
Should I block /wp-admin/?
Yes, and almost every WordPress install already does. What you should not block is /wp-content/ or /wp-includes/, which is advice that circulated in the early 2010s. Googlebot renders pages, and blocking the CSS and JavaScript means it renders yours broken and judges it on that.
Do I need a robots.txt at all?
Not necessarily. A 404 on /robots.txt is valid and means everything is crawlable. The single reason to add one to a small site is the Sitemap line, which is read by every crawler including ones you have never verified with.
Why does it say Googlebot ignores my rules?
Because there is a group specifically naming Googlebot somewhere in the file. A crawler obeys the most specific group that matches its name and disregards all the others, including the wildcard. Every rule you want Googlebot to follow has to be repeated inside that group.
Where to go from here
- Robots.txt Generator — Build a corrected file and test it before you upload it.
- XML Sitemap Checker — Check the sitemap this file declares actually works.
- Google Crawler Simulator — See what a crawler gets once it is allowed through.
- The thinking behind the SEO audit checklist — The crawl and index checks in context.
The guide behind this tool
Built by GoDesign FZE, who run technical SEO fixes for UAE sites.
Get in touch
Want someone to just do this part?
The tool is free and stays free. If you would rather hand the work over, tell us what you are dealing with and you get a straight answer on scope, cost and timeline.
- You own everythingRepository, hosting and domain credentials sit in your name from day one, not handed over at the end.
- A written timelineMilestone dates are agreed in writing before work starts, so you always know what ships next.
- One business dayEvery enquiry gets a reply from the person who would scope the work, not a sales sequence.
Media City, Dubai, UAE · DHA Phase 2, Islamabad, Pakistan
Would rather type than fill in a form?
Message on WhatsApp