Robots.txt & XML Sitemaps: The Complete Guide
robots.txt and XML sitemaps are the two plain-text files that tell search engine crawlers where they're allowed to go and where your important pages actually are — robots.txt sets crawl permissions, and a sitemap is a direct list of URLs you want indexed. This guide covers the directives that actually work, the one most people get backwards (blocking a crawl is not the same as blocking an index), and the mistakes that quietly tank organic traffic for weeks before anyone notices.
What robots.txt actually controls
robots.txt is a plain-text file at the root of a domain (/robots.txt) that every well-behaved crawler checks before requesting anything else on the site. It's made of groups, each starting with a User-agent line and followed by Allow / Disallow rules that apply to that user agent:
User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /admin/help.html
Sitemap: https://example.com/sitemap.xmlThree things matter here that a lot of hand-written robots.txt files get wrong:
- It's a crawl instruction, not a security or privacy control. robots.txt is publicly readable by definition — anyone, not just search engines, can see exactly which paths you've listed as sensitive. Never rely on it to hide a URL that genuinely needs to stay private; that's what authentication is for.
- The most specific rule wins, for Google. If
Disallow: /admin/andAllow: /admin/help.htmlboth match a URL, Googlebot uses whichever pattern is longer (more specific) — not whichever line appears first or last in the file. Crawl-delaydoes nothing for Google. Googlebot ignores it entirely; it's a Bing/Yahoo-era directive. Managing Googlebot's crawl rate is done in Search Console, not robots.txt.- It says what may be fetched, not what your site is about. That's the job of llms.txt, a separate and newer Markdown file that gives AI assistants a summary of your site and a curated list of its best pages. It controls nothing, so AI crawler access still belongs in robots.txt; the llms.txt Generator builds and validates one.
The mistake almost everyone makes: crawl-blocking vs. index-blocking
Disallowing a URL in robots.txt stops crawling — Google won't fetch the page's content. It does not stop indexing. If enough other pages link to a disallowed URL, Google can still list it in search results using just the anchor text and surrounding context from those links, typically showing "No information is available for this page." instead of a real snippet.
If the actual goal is "keep this page out of search results entirely," the correct tool is a noindex meta tag or X-Robots-Tag: noindex HTTP header — and critically, the page must not be blocked in robots.txt, because a crawler that can't fetch the page can't see the noindex instruction either. This is the single most common way sites end up with the exact opposite of what they intended: a robots.txt block that was supposed to hide a page instead leaves it indexed with a blank snippet, because the noindex tag on the page itself was never seen.
The catastrophic one-liner
This is the highest-stakes mistake robots.txt allows, and it happens more often than you'd expect — usually a staging robots.txt pushed to production by accident:
User-agent: *
Disallow: /That blocks every crawler from every page on the entire site. Traffic doesn't drop instantly — it fades as Google gradually stops recrawling pages it's no longer allowed to fetch, which makes it a slow, quiet failure that's easy to miss until weeks of organic traffic are already gone. It's exactly the kind of mistake worth checking for after any deploy that touches server config, not just after a robots.txt edit specifically — this is what Frost Rank's Robots.txt / Sitemap Validator flags first.
XML sitemaps: the other half of crawl control
Where robots.txt says where crawlers can't go, a sitemap is a positive signal — a direct, structured list of URLs you actually want indexed, plus metadata like last-modified date:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/</loc>
<lastmod>2026-09-01</lastmod>
</url>
</urlset>A sitemap is a suggestion, not a guarantee — Google still decides independently whether to crawl and index each URL. But it's a genuinely useful signal on large or poorly-linked sites, since it's the most direct way to surface pages that internal linking alone might leave hard to discover.
Two hard limits: 50,000 URLs or 50MB uncompressed per file, whichever comes first. Past that, use a sitemap index file — a sitemap that lists other sitemaps instead of pages — and reference the index (not each individual file) from robots.txt with a Sitemap: line. Because sitemaps are XML, a single malformed tag can break parsing for the entire file, unlike robots.txt's plain-text format where a bad line just gets ignored — which is why XML sitemap errors tend to be all-or-nothing rather than partial.
| robots.txt | sitemap.xml | |
|---|---|---|
| Signal type | Negative (what crawlers can't fetch) | Positive (what you want indexed) |
| Format | Plain text | XML |
| Bad line behavior | That line is ignored, rest still works | Whole file can fail to parse |
| Hard limit | None | 50,000 URLs / 50MB per file |
| Referenced from | — | Sitemap: line in robots.txt |
Checking both files together
Because a robots.txt mistake and a sitemap mistake often mask each other — a sitemap can dutifully list a URL that robots.txt is quietly blocking — it's worth validating both from the same pass rather than checking them separately. Frost Rank's Robots.txt / Sitemap Validator handles both file types and can fetch either directly from a live URL, and pairs well with the Redirect Chain Checker and HTTP Header Checker for the rest of the technical-crawl picture — what actually happens once a crawler is allowed to request a URL.
Frequently asked questions
No. A Disallow stops crawling, not indexing, so a linked page can still appear without a snippet; use noindex instead, and don't block that page in robots.txt. See crawl-blocking vs. index-blocking explained.
Yes — robots.txt rules are scoped per exact origin (scheme + host + port). https://example.com/robots.txt and https://www.example.com/robots.txt are two entirely separate files as far as crawlers are concerned, even if your redirects make them look like the same site to a human visitor. If you serve both variants without redirecting one to the other, both need their own correct robots.txt.
Usually unnecessary on its own — pages with no inbound links and no real content rarely get indexed regardless. It's not harmful either, as long as the disallow pattern is scoped tightly and doesn't accidentally catch something you need crawled (a common mistake: blocking /wp-admin/ is fine, blocking /wp-a is not, since that pattern also matches paths like /wp-articles/).
A sitemap lists page URLs, up to 50,000 per file; a sitemap index lists other sitemaps, for sites that outgrow one file. See the sitemap limits.
Not for Google, which applies the most specific (longest) matching rule wherever it appears. Some other crawlers use the first match instead. See how Google matches rules.
Yes — any line starting with # is a comment and ignored by crawlers. Useful for leaving a note about why a particular Disallow line exists, since robots.txt files tend to accumulate rules nobody remembers the reason for.
Written from real Search Console cleanup work — including the exact "Disallow: /" staging-to-production accident and the crawl-vs-index confusion described above, both seen repeatedly across real client audits.
Was this guide helpful?
Rate it and leave a comment — takes 10 seconds, and genuinely helps decide what to fix or build next.