AI Training Opt-Out: One robots.txt Rule Blocks Four

On October 30, 2026, the four companies whose models most site owners worry about — Google, OpenAI, Microsoft and Anthropic — agreed to honour a single robots.txt rule that opts a site out of AI training. The rule is two lines long, it lives in the file you already have, and it replaces the pile of per-company crawler blocks that publishers have been maintaining one vendor at a time. The token is __usage_ai-training, and the short version of the whole story is this: add it to your robots.txt, and — once each company's rollout lands — the four companies will treat your content as off-limits for training their generative models, without touching your search rankings. The longer version matters, though, because the rule has limits that are easy to misunderstand — it is a draft, not a finished standard; it only looks forward, never backward; and it says nothing at all about whether your pages show up in AI answers. This article covers the rule itself, the rollout dates, the four things it will not do, and the decision of whether to use it at all.

What the new rule actually is

Until now, opting out of AI training meant maintaining a separate robots.txt group for every company's training crawler: GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google, and a mix of tags and crawler rules for Microsoft. Every new model vendor added another line to maintain, and a site owner who missed one vendor's announcement stayed opted in by default. The new approach changes what the User-agent line names. Instead of naming a crawler, it names a purpose: AI training. The proposal is an IETF Internet-Draft written by Kevin Kelley, Anthropic's technical lead for standards, titled "Machine-Readable Training Preference Signal," and it introduces one policy token, __usage_ai-training, that any participating company recognises.

The rule itself is deliberately boring, which is the point. It uses no new directive names and no new grammar — just the ordinary robots.txt groups defined by RFC 9309, the Robots Exclusion Protocol. To opt an entire site out of AI training, you add this group to your robots.txt:

User-agent: __usage_ai-training
Disallow: /

That is the whole mechanism. A crawler that understands the token reads the group as a statement about use, not about crawling: the content under those paths should not be used to train generative AI models. And because it is ordinary robots.txt syntax, every existing parser keeps working — a parser that does not recognise the token simply ignores the group the way it ignores any group for an unfamiliar crawler. Nothing breaks, nothing needs a new file, and there is no vendor dashboard to configure.

How to use it: the full opt-out, and the partial one

Most sites that opt out will want the site-wide version above. But the token also supports partial opt-outs, because the normal robots.txt path rules still apply: the longest matching rule wins, so an Allow line for a specific folder carves that folder back out of the block. The draft's own example is a site that blocks training everywhere except its press kit:

User-agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Worked example: imagine you run a news site. You might block training on your reporting, where the value is the journalism itself, while allowing it on your press releases and public statements, which you want widely reused. Or a documentation site might block its tutorials but leave its API reference open. The point is that the decision is per path, not just per site, and it uses the same matching rules you already know from ordinary robots.txt work — Google's robots.txt documentation describes how its crawlers group and match these rules. If you are unsure how your file parses, run it through a free robots.txt and sitemap checker before deploying — a typo in a Disallow pattern is silent, and the failure mode is "the rule quietly does not cover what you thought it did."

Three placement rules matter. First, the group belongs in the same robots.txt at your domain root that your search rules live in; it does not need its own file. Second, keep your existing per-company rules in place alongside it for now — the new token is additional, and older crawlers that do not yet support it still follow the old rules. Third, remember that the wildcard group does not cover this token: the draft states explicitly that User-agent: * must not be interpreted as matching the policy token, so a file that only has a * group expresses no training preference at all. You have to write the token's name out.

When each company actually starts honouring it

The announcement was joint, but the rollouts are not simultaneous. That gap matters for expectations: adding the rule today does not mean all four companies respect it tomorrow. Here is where each one stands, as reported at the announcement:

The staggered dates are also a reason to add the rule sooner rather than later, even though enforcement is months away for two of the four: crawlers act on the file they fetch when they fetch it, and a rule added in November is already in place when Microsoft's February date arrives and when Google's early-2027 rollout begins. There is no application to file and no waiting list — the file itself is the instruction.

Four things this rule will not do

The announcement is easy to over-read, so here are the four limits, stated plainly.

1. It will not remove anything already collected. The token governs future crawls only. Content that was already gathered and folded into existing training sets stays there; no robots.txt rule can reach back into a finished model and extract it. If your concern is material that is already out in the world, this rule does not solve it — it only changes what happens next.

2. It will not block search indexing. This is the confusion to avoid. The token expresses a preference about training use, and the draft keeps usage preferences and crawl rules independent and orthogonal. Googlebot and Bingbot keep crawling and indexing your pages for ordinary search results exactly as before. Blocking AI training does not cost you search traffic, and — equally important — it does not hide you from search either. If your actual goal is to keep a page out of search results, that is a different job, covered by noindex, and our robots.txt and sitemap guide explains why confusing the two backfires.

3. It will not stop your content appearing in AI answers. Training a model and grounding an answer are different uses. The token covers the first: using your content to modify a model's learned parameters. It does not cover retrieval at answer time, when a system fetches your live page to inform a response. Microsoft makes this split explicit with a separate AI Grounding preference, expressed through page-level Bingbot controls rather than robots.txt, that governs whether a page can be pulled into answers in Bing, Copilot and Microsoft 365. Google has its own wrinkle: if you use the common token without explicitly mentioning Google-Extended, Google plans to read it as a Google-Extended disallow — and publishers who specifically allow Google-Extended remain eligible to appear in Gemini Apps responses and receive clicks from them. Training opt-out and answer visibility are two levers, not one.

4. It is a draft, not a finished standard, and it is voluntary. The document is an individual Internet-Draft with an expiry date of May 3, 2027; it has not received working-group endorsement, let alone RFC ratification, and its syntax could still change before it becomes a formal standard. The four companies have committed to honour it, which gives it practical force, but it remains a polite, machine-readable request rather than a legal shield. Nobody has independently tested enforcement at scale yet, and private agreements can override it. Treat it as the best available standard mechanism, not as a guarantee.

Should you opt out? A decision, not a default

The honest answer is that it depends on what your content is worth to you versus what AI visibility is worth to you — and the two are not the same calculation for every site. AI crawler blocking makes most sense when your content's value is the content itself: original reporting, proprietary research, paid course material, photography, or anything where a model trained on it becomes a substitute for visiting you. It makes least sense as a reflex, because opting out of training does not buy you anything in search, and it does not, on its own, change whether AI answers cite you — that is governed by the separate grounding and retrieval controls above, plus whether your pages are built to be cited at all.

Two questions settle it for most sites. First: if a future model can answer your readers' questions without them visiting you, do you lose something real — subscriptions, sales, ad impressions, attribution? If yes, opt out, at least for the sections where that is true. Second: do you actively want AI systems to find and summarise your pages, because discovery through AI answers brings you readers? If yes, do not confuse the two files: keep training preferences in robots.txt and discovery signals in llms.txt, the Markdown file that tells AI assistants what your site is about and which pages matter most — see our breakdown of llms.txt for AI discovery. The two files do different jobs, and a site can reasonably use both: opt out of training while still making itself easy to cite.

One more practical note for anyone who already maintains per-company rules: do not delete them yet. The new token is the future common path, but GPTBot, ClaudeBot and Google-Extended rules are what actually bind crawlers today, during the months before each company's rollout completes. The draft itself lets companies treat their own proprietary tokens as more specific than the common one, so a site can set the broad rule and still carve out an exception for one company. Add the new group, keep the old ones, and revisit in mid-2027 when all four rollouts are live.

What AI training opt-out tools can and can't check

As the token spreads, expect a wave of checkers that claim to verify your opt-out. A robots.txt checker can genuinely verify three things: that the file is reachable at your domain root, that the __usage_ai-training group parses correctly, and that your Allow and Disallow patterns cover the paths you intend — including the classic mistakes, like a pattern that accidentally matches more than you meant. What no checker can verify is the other side of the deal: whether a given company's crawler actually honoured the preference on its last visit. That is observable only indirectly, through crawler logs and the companies' own statements. So use a checker to validate the file, and use your server logs to confirm which training crawlers still visit and how often — the file is your half of the contract, and the logs are the only evidence about theirs.

Frequently asked questions

Does one robots.txt rule really block all four companies from AI training?

Once each company's rollout is complete, yes — Google, OpenAI, Microsoft and Anthropic have all committed to honour the __usage_ai-training token. But rollouts are staggered: Anthropic is already implementing it, OpenAI follows over the coming months, Microsoft enforces from February 1, 2027, and Google plans an early-2027 rollout. Until a company's rollout lands, its older per-crawler rules are what bind it.

Will blocking AI training hurt my Google rankings?

No. The token expresses a preference about training use only, and it is independent of crawl rules. Googlebot and Bingbot keep crawling and indexing your pages for ordinary search results exactly as before. If your rankings change after adding the rule, something else changed.

Can I block AI training for most of my site but allow it for some pages?

Yes. Normal robots.txt path matching applies: Disallow: / with Allow: /press-kit/ blocks training everywhere except that folder. The longest matching rule wins, just as with ordinary crawl rules.

Do I still need GPTBot, ClaudeBot and Google-Extended rules?

For now, yes. Keep them alongside the new token until all four rollouts are complete — the old rules are what bind crawlers during the transition, and companies may treat their own tokens as more specific than the common one. Revisit in mid-2027.

Does this remove my content from models that already trained on it?

No. The token only governs future crawls. Content already collected and incorporated into existing training sets is unaffected, and no robots.txt rule can extract it from a finished model.

Does opting out of training also keep my pages out of AI answers?

Not necessarily. Training and answer-time retrieval are different uses. Microsoft has a separate AI Grounding preference for answers in Bing, Copilot and Microsoft 365, and Google treats the common token as a Google-Extended disallow, which affects Gemini Apps eligibility. Check each company's grounding controls separately if answer visibility is your concern.

The old way of opting out asked every publisher to track every vendor's crawler list. The new way asks for two lines in a file you already own. Add the token, keep your existing rules until the rollouts land, validate the file, and make the training-versus-visibility decision deliberately — section by section, not by reflex. The draft may still change before it becomes a standard, but the direction is set: one rule, four companies, and a clear line between training your replacement and earning a citation.