Robots.txt / Sitemap Validator
A robots.txt checker and sitemap validator scans these two crawl-control files for the specific mistakes that actually block Google from indexing your site. Paste your robots.txt or sitemap.xml — or fetch it directly from a URL — and catch crawl-blocking mistakes before Google finds them for you.
1 Paste or fetch your file
Fetching a URL routes through our server to work around browser restrictions most sites impose on cross-site requests — the fetched content isn't stored or logged. Pasting content directly stays fully in your browser.
2 Validation results
Why robots.txt and sitemap mistakes are so costly
robots.txt is the first file most crawlers check before touching anything else on your site — one wrong line can accidentally block Google from crawling your entire site, and you often won't notice until organic traffic quietly drops over weeks. It's plain text with no validation built in, so a typo doesn't throw an error; it just silently does the wrong thing.
Your sitemap.xml has the opposite problem: it's XML, so a malformed tag breaks the whole file, and search engines simply stop reading it. Both files are easy to get right and easy to get catastrophically wrong, which is exactly why this validator checks for the specific mistakes that actually show up in Google Search Console — not just whether the syntax is technically parseable. Once your crawl-control files are clean, the Redirect Chain Checker and HTTP Header Checker cover the other side of technical SEO: what happens once a crawler actually requests a URL.
For example, this two-line robots.txt:
User-agent: *
Disallow: /blocks every crawler from every page on the site — exactly the kind of catastrophic, easy-to-miss mistake this tool flags immediately.
How to use this tool
-
1
Choose the file type
Select robots.txt or sitemap.xml at the top of the tool.
-
2
Fetch or paste the file
Enter the file's full URL and click "Fetch & check" to load the live version, or paste the contents straight into the box.
-
3
Review the results
Errors, warnings and notices are listed with the line number where one applies, and each explains what the problem does.
-
4
Fix the file
Update the file on your server or in your CMS, starting with anything marked as an error.
-
5
Re-check and resubmit
Fetch the file again to confirm the fix, then resubmit the sitemap in Google Search Console.
Common use cases
- Checking robots.txt straight after a launch or migration, when a staging site's "Disallow: /" most often slips into production.
- Auditing a client's or competitor's crawl rules as part of an SEO audit.
- Validating a sitemap generated by a plugin or custom script before submitting it.
- Confirming that a sitemap index lists its child sitemaps correctly.
- Troubleshooting a "Sitemap could not be read" or "Couldn't fetch" error in Search Console.
Tips and common mistakes
- Blocking a crawl does not block indexing. A URL disallowed in robots.txt can still appear in results if other sites link to it. To keep a page out, leave it crawlable and add a noindex tag, which the Meta / OG Tag Generator can produce.
- Don't block CSS or JavaScript. Search engines need them to render the page the way visitors see it.
- Paths are case-sensitive prefixes.
Disallow: /Admindoes not block/admin, andDisallow: /blogalso blocks/blog-archive. - List only final, indexable URLs in a sitemap. Redirects, noindex pages and 404s waste crawl attention; check doubtful entries with the Redirect Chain Checker. The robots.txt and XML sitemaps guide covers the rules in depth.
Frequently asked questions
Combined with "User-agent: *", it tells every crawler not to index anything on your site. It's almost always left over from a staging environment that got pushed to production by accident — and it's the single most common catastrophic robots.txt mistake.
No. Googlebot ignores Crawl-delay entirely — it only affects Bing and a handful of other crawlers. If you need to manage Googlebot's crawl rate, that's done in Search Console instead, not in robots.txt.
50,000 URLs, or 50MB uncompressed, whichever comes first. Past that, you need a sitemap index file that references multiple smaller sitemaps.
Yes — use the "Fetch & check" button and enter your URL. It routes through Frost Rank's server to bypass the browser CORS restrictions that block most direct cross-site requests; nothing fetched that way is stored or logged.
Yes, completely free with no signup. Pasted content is validated entirely in your browser — the one exception is the optional URL-fetch feature, documented above.
Checks for the crawl-blocking mistakes that actually show up in Google Search Console — an accidental "Disallow: /" left over from a staging environment, relative Sitemap URLs, and CSS/JS blocks that break rendering-based indexing — not generic spec pedantry.
Was this tool helpful?
Rate it and leave a comment — takes 10 seconds, and genuinely helps decide what to fix or build next.