OpenAI, Google, and Microsoft will support the __usage_ai-trainin… identifier in robots.txt

On October 30, 2026, OpenAI, Google, and Microsoft announced plans to support a new mechanism for specifying the use of web content for AI training via robots.txt. Site operators can use the identifier __usage_ai-training to communicate permission or refusal for training use to multiple AI providers at once. The aim is to reduce the burden of managing settings for each provider individually.

The timing of support and how the new specification is handled when used alongside existing settings differ by provider.

OpenAI: Making it easier for websites to express AI training preferences

Google: Supporting a shared AI training control for websites

Microsoft: Your content, your choice: clearer preferences for training and grounding

Specifying permission/refusal for training use via robots.txt

In the mechanism that OpenAI, Google, and Microsoft plan to support, robots.txt, the file that contains instructions for crawlers, is used.

To communicate whether content may be used for training, site operators write __usage_ai-training in the User-Agent line of robots.txt. This is an identifier indicating a specification regarding training use, rather than the name of a specific crawler. For example, to refuse the use of content across the entire site for generative AI training, specify it as follows.

User-Agent: __usage_ai-training Disallow: /

Disallow indicates refusal of training use, and / indicates the entire site. To permit training use only for specific paths, use Allow. For example, to refuse across the entire site but permit use only for content under /press-kit/, write it as follows.

User-Agent: __usage_ai-training Disallow: / Allow: /press-kit/

What these settings communicate is whether content may be used for training. Whether crawlers may crawl the site is specified separately under existing crawler rules.[1]

This mechanism is based on the draft specification "Machine-Readable Training Preference Signal" published on October 30, following discussions in the IETF AI Preferences (AIPREF) working group. It is a draft submitted by Kevin Kelley of Anthropic and is not a standard approved by the IETF.

Each company to support it in turn; Anthropic also begins implementation

OpenAI aims to implement it within the next few months and will provide details when it supports it. Google plans to support it in early 2027. Microsoft will begin applying it from February 1, 2027, to sites and pages whose settings are read during crawling.

As for Anthropic, Kelley explained in a post to the IETF public mailing list on October 30 that it had begun implementation. He said it will take time for each service to support it and did not mention a completion date.[2]

OpenAI continues existing crawler settings

At OpenAI, existing settings for crawlers will continue to apply. Instructions for the training crawler GPTBot and the search crawler OAI-SearchBot can still be set separately, allowing a site to refuse training use while permitting inclusion in ChatGPT search results.[3]

For example, to refuse crawling by the training crawler GPTBot and permit crawling by the search crawler OAI-SearchBot, write the following in robots.txt.

User-Agent: GPTBot Disallow: /

User-Agent: OAI-SearchBot Allow: /

Google controls training and grounding together

At Google, the existing Google-Extended is used to collectively allow or deny both the use of content for training and its use for grounding, where the content is referenced as a source when generating answers. This covers training of future Gemini models and grounding in the Gemini app and in Grounding with Google Search on Vertex AI.

Once Google begins reading the specification via __usage_ai-training, how a refusal of training use is handled will depend on whether Google-Extended is specified separately.

For example, if robots.txt is written as follows and Google-Extended is not specified separately, Google will treat this as a refusal via Google-Extended. As a result, both the training and grounding described above will be restricted.

User-Agent: __usage_ai-training Disallow: /

On the other hand, even if training use is refused via __usage_ai-training, if Google-Extended is separately allowed as follows, both training by Google and use for grounding will be permitted.

User-Agent: __usage_ai-training Disallow: /

User-Agent: Google-Extended Allow: /

According to Google's explanation, sites that explicitly permit use via Google-Extended will be eligible to be referenced in answers in the Gemini app. Google-Extended settings do not affect inclusion in Google Search or search rankings.

Microsoft configures training and grounding separately

Microsoft will allow separate configuration of the use of web content collected by its crawlers for training and its use as a source when generating answers. The training-use setting is called AI Training, and the answer-reference setting is called AI Grounding.

Training use can be specified for an entire site or specific paths via __usage_ai-training in robots.txt. It can also be specified with per-page meta tags, and meta tag specifications take precedence over robots.txt settings. Meanwhile, use for grounding is not specified in robots.txt but is specified with per-page meta tags.[4]

For example, to refuse training use of a certain page while permitting reference when generating answers, write the following meta tag inside the head element of that page's HTML. name="Bingbot" indicates a specification directed at Microsoft's crawler, Bingbot.

Changing the grounding specification from allow to disallowextended allows reference in services operated by the company, such as Copilot and Microsoft 365, while refusing use in grounding services the company provides to third parties.

At Microsoft, if no applicable settings exist, both training and grounding are treated as permitted. Even if training or grounding is refused through these settings, the site will not be removed from being eligible for inclusion in ordinary Bing search results.

Column: Migration is required to maintain restrictions under Microsoft's old settings

Until now, Microsoft has restricted the scope of use of a page's content for training and grounding according to NoArchive or NoCache specifications written in the page's meta tags. From March 1, 2027, use restrictions based on these specifications will no longer apply, so to continue restrictions, it is necessary to migrate to the new settings before that date.

However, once Microsoft reads the new AI Training or AI Grounding settings within a site, application of the old settings for that site ends without waiting for March 1.

Even if the new settings are written on a page, they will not be reflected until Microsoft reads them via recrawling, so to maintain grounding restrictions, it is also necessary to register temporary settings in Bing Webmaster Tools. Registration of temporary settings begins on December 15, 2026, and is available to sites using NoArchive or NoCache as of that date.

Temporary settings registered in Bing Webmaster Tools apply until the new settings written on each page are read via recrawling. However, even if they are not read, they expire on September 30, 2027. Specific specification methods and migration conditions can be confirmed in Microsoft's migration guide.