Enter a path (e.g.
/admin/page) or a full URL. Only the path portion is used.
Generated robots.txt
Live
Robots.txt Tester
Paste your robots.txt content to validate its syntax, check for common issues,
and test whether specific URL paths are allowed or blocked for each crawler.
Use the Generate tab to build a clean robots.txt from scratch.
Everything runs in your browser — nothing is sent to a server.
What this tool does
The Validate tab parses your robots.txt and breaks it into User-agent blocks.
It checks for syntax errors, rules that disallow all crawlers (Disallow: /),
missing Sitemap references, and rules that specifically block Googlebot.
Each issue is flagged with a colour-coded badge (green = OK, yellow = warning, red = error),
and a summary shows the total number of rules, warnings, and errors found.
The Test a URL tab lets you paste your robots.txt and enter any
URL path to see exactly whether it would be allowed or blocked — and which specific rule is responsible.
The Generate tab builds a clean robots.txt from checkboxes with a live
preview you can copy with one click.
How to use it
- Validate: Paste your
robots.txtcontent into the textarea and click Validate. Review the per-block breakdown and the warnings/errors list. - Test a URL: Paste your
robots.txtcontent, then type a path such as/admin/settingsand click Test URL. The result shows allowed or blocked, along with the matching rule. - Generate: Tick the rules you want, enter your sitemap URL, and copy the live-generated output directly into your server's
robots.txtfile.
Frequently Asked Questions
Does robots.txt actually prevent crawlers from accessing pages?
No — robots.txt is a voluntary standard. Well-behaved crawlers like Googlebot respect it,
but malicious bots, scrapers, and bad actors can ignore it entirely. If a page must not be publicly accessed,
protect it with authentication or a firewall. A common mistake is using robots.txt alone to hide
sensitive content; that is not reliable security.
What is the difference between Disallow: / and Disallow: /admin?
Disallow: / blocks every URL on the site for that User-agent — including the homepage.
This is almost never intentional and will prevent Google from indexing any of your pages.
Disallow: /admin blocks only URLs that start with /admin, leaving the rest of the
site crawlable. This tool flags Disallow: / as an error when it appears under User-agent: *
or User-agent: Googlebot.
Do I need to list every crawler separately?
No. The User-agent: * block acts as a catch-all for any crawler not explicitly named.
You only need separate blocks when you want to give a specific bot different rules from the default —
for example, allowing Googlebot access to a section that is blocked for all others. Most sites only need
a single User-agent: * block plus a Sitemap: line.