Google to Honour a Shared AI Training Opt-Out in robots.txt

Google to honour a shared AI training opt-out in robots.txt

Released on 30 October 2026, Google's announcement sets out plans to read a new cross-provider robots.txt token, __usage_ai-training, as an AI training opt-out alongside its own Google-Extended control. Read the Google Search Central announcement here.

Until now, a site owner who wanted to keep their content out of AI training has had to write a separate robots.txt rule for each company: Google-Extended for Google, GPTBot for OpenAI, ClaudeBot for Anthropic, and another line every time a new model provider starts crawling. The proposal Google now backs replaces that growing list with a single group. A site names one shared "policy token", sets its preference once, and every AI provider that adopts the token reads the same instruction.

Nothing changes yet. Google's crawler documentation, updated the same day, says it intends to support the signal "in early 2027", and the blog itself calls the scheme a "draft proposal". What matters now is the detail of how Google will interpret the token, because it has a knock-on effect for any business that wants to be cited in Gemini, and that is worth understanding before anyone edits a robots.txt file.

How it works

The mechanism comes from an Internet-Draft published at the IETF on 30 October, draft-kelley-aipref-training-preference-signal, submitted by Kevin Kelley of Anthropic. It adds no new directives to robots.txt. Instead it reuses the grammar of RFC 9309, the robots.txt standard, and treats __usage_ai-training as if it were a user agent. A Disallow rule under that token expresses a preference against AI training for the matching paths, and an Allow rule expresses a preference in favour. Rules can be set per path, so the draft's own example opts a whole site out except its press kit:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Three rules in the draft change how existing files will be read. First, a User-Agent: * group must not be treated as matching the token, so a blanket rule written for all crawlers says nothing about training. Second, if no group names the token, no preference has been expressed at all. Third, the token is "not a directive for crawling behavior": it states what content may be used for, not whether it may be fetched, and crawl limits still belong in each crawler's own group. "AI training" takes its meaning from the IETF AI Preferences working group's vocabulary draft: using content to modify the learned parameters of a generative AI model. Models used only for classification, ranking or scoring fall outside that definition.

Google's side is set out in two places. The blog says that when a site uses the token to opt out "and Google-Extended is not explicitly specified", Google "plans to interpret that as if the publisher had disallowed Google-Extended". Company-specific controls carry on working, and the draft lets a provider treat its own token as more specific than the shared one, so a file with its own Google-Extended group will be read on that group. The Google-Extended entry in Google's crawler documentation then defines what a disallow covers: training future Gemini models for Gemini Apps and the Vertex AI API for Gemini, and grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google-Extended has no user agent string of its own, and Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal".

Examples and use cases

Start with a small UK brand whose robots.txt has never mentioned AI. Under the draft it expresses no training preference today, and it will still express none in 2027 unless someone adds a group for the token. The arrival of the shared control does not opt anyone out by default. Silence remains silence, and each provider's existing policy applies.

Now take a business that does not want its content used to train models but does want to be cited when Gemini Apps answer a question. If it adds only the shared opt-out, Google's mapping turns that into a Google-Extended disallow, and Google-Extended governs Gemini Apps grounding as well as training. Our reading of the two Google sources together is that such a site is likely to lose Gemini Apps grounding eligibility too. Google's blog says that sites which "specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps", so staying eligible needs a file shaped like this:

User-Agent: __usage_ai-training
Disallow: /

User-Agent: Google-Extended
Allow: /

That file tells participating providers "no training" and tells Google that Gemini may use the site, training included. Google-Extended is a single switch, so there is no combination that gives Google grounding without training.

A third case is the site that already lists GPTBot, ClaudeBot and Google-Extended one by one. Those rules keep working. The shared token would let the owner state the same broad preference once for any provider that adopts it later, with the named groups kept as the precise, provider-by-provider controls. Per-path rules also open up a middle ground that most sites have never used: a retailer could allow training on product and press pages it wants widely known, while disallowing it on paid guides or original research that it sells.

What this means for UK site owners

The legal backdrop has not moved. The government's March 2026 report on copyright and AI, published under the Data (Use and Access) Act 2025, says the broad data mining exception with an opt-out that it originally preferred was rejected by most respondents to its consultation, which drew 11,520 responses. It will not introduce reforms to copyright law until it is confident they meet its objectives, and on technical tools it proposes to keep the need for regulation under review while supporting market-led standards. In the UK, then, a robots.txt training preference is a signal that AI companies choose to honour, not an opt-out written into law. Google ties its proposal to the EU's Code of Practice for general-purpose AI model providers, which recognises robots.txt as a primary way for publishers to express machine-readable preferences. The UK has no equivalent rule today.

Plenty of sites already use robots.txt this way. The same report, citing Cloudflare Radar's 2025 review, notes that AI crawlers were the most frequently fully disallowed user agents found in robots.txt files. A shared token mainly makes that choice cheaper to express and harder to get wrong as new providers appear.

For search visibility, the most useful line is the one about what does not change. Google's AI features documentation says robots.txt rules for Googlebot are the control for Search, and that limiting what appears from your pages in AI Overviews or AI Mode is done with the ordinary preview controls: nosnippet, data-nosnippet, max-snippet or noindex. Google-Extended, and therefore the shared token as Google reads it, sits outside Search. A business can opt out of Gemini training without affecting its Google rankings, and it cannot opt out of AI Overviews this way.

The real decision is commercial, and it is made provider by provider. Our llms.txt server-log study sets out which bots each AI company runs for search, live fetches and training, and the AI search optimisation playbook covers how to keep the search bots in while you make a separate call on training. If being cited in AI answers is part of your growth plan, that call belongs inside a wider generative engine optimisation strategy rather than a one-line robots.txt change made in isolation.

Practical considerations

Treat everything here as provisional. The IETF document is an individual Internet-Draft: the datatracker marks it "not endorsed by the IETF" with "no formal standing", and it expires on 3 May 2027 unless it is updated. The token name, the precedence rules or the definition of training could all change before Google switches support on, and Google says it will "provide updates on our progress". Nor is there yet a list of adopters. Google says other providers "can handle it similarly", but the crawler documentation pages published by OpenAI, Anthropic and Perplexity made no mention of the token when we checked on 2 November. Adding the group now is harmless to parsers that do not recognise it, because they ignore unknown groups, but bear in mind that Google will start reading it as a Google-Extended disallow once support begins.

Remember too that robots.txt is a request, not a lock. It does not stop a crawler fetching a page, the government's report records concerns about unlabelled "stealth crawling" that sidesteps it, and a CDN or firewall setting can block bots your file welcomes. Before 2027, check three things: what your robots.txt says to each named AI bot today, whether any rule under User-agent: * was meant to cover AI training (under the draft it will not), and whether your CDN's bot settings agree with the file. Do not expect to see Google-Extended in your server logs either. Google crawls with its normal user agents and uses the token only as a control.

FAQ