OpenAI announces it will follow robots.txt shared AI training preference signal

Oct 31

1. 19:30

   Unite.AI

   Google plans to support shared AI training control signals in robots.txt

   On October 30, Google announced plans to support shared AI training control signals in robots.txt, allowing website owners to express training preferences to multiple AI vendors in a unified way, and said the goal is to implement it by 2027. The mechanism is based on the work of the IETF AI Preferences working group, writing a __usage_ai-training policy token in the User-Agent line, without adding new directive names or syntax; opt-out declarations that do not specify Google-Extended will be treated as prohibiting its use. OpenAI committed the same day to honor the signal and plans to implement it within the coming months, while Microsoft announced that starting February 1, 2027, it will replace NoArchive and NoCache with the new preference.

   Expand summary
2. 18:08

   Testing Catalog

   October 31 AI Daily Brief: Anthropic subscription renamed, Grok Bot 1.0 released

   Anthropic renamed Max 5x to Max ($100) and Max 20x to Ultra ($200), and the verified Ultra version removed the 5-hour limit; the maximum request size for the Claude API increased from 32 MB to 128 MB. The first stable version of xAI's Grok Bot 1.0 is available for download, with new per-bot voice and voice language settings. NVIDIA open-sourced the 9.6M-parameter voice cleanup model RE-USE, and Pika released the open-weight video model H3 Flex.

   Expand summary
3. 10:09

   Jason ZhuRelated discussion

   Jason Zhu joined this discussion ↗

   This post contributes to heat; it is not a formal report or confirmed fact.

4. 10:00

   SiliconANGLE

   Google, Microsoft, Anthropic and OpenAI to support new AI crawler opt-out standard

   Google, Microsoft, Anthropic and OpenAI announced they will support an in-development website crawler standard that lets website operators block all crawlers collecting training data for generative AI models with a single robots.txt entry; the setting does not apply to models performing only non-generative tasks. The four companies have submitted a draft to the IETF, which maintains the Robots Exclusion Protocol, and Microsoft will also provide a setting called AI Grounding to block web access requests made by models to answer user questions. Microsoft and OpenAI plan to support the standard within the next few months, Google will follow next year, and Meta said it supports the idea but has not yet committed to implementing it.

   Expand summary
5. 06:11

   Thurrott

   Anthropic, Google, Microsoft and OpenAI agree to let websites opt out of AI training

   Anthropic, Google, Microsoft and OpenAI have agreed to support and implement an opt-out mechanism developed by an IETF working group, allowing website owners to express their preferences regarding the use of their content for generative AI training, separately from search indexing and archiving. OpenAI calls it a voluntary mechanism that lets publishers convey the same preference to multiple participating vendors at once; Microsoft says publishers should not have to choose between Bing discoverability and models using their work. Anthropic has already implemented the mechanism, while Google, Microsoft and OpenAI will implement it in 2027, after which it will be proposed as an international standard.

   Expand summary
6. 04:26

   The Verge AI

   OpenAI, Google, Microsoft, and Anthropic will adopt a unified AI crawling opt-out standard

   OpenAI, Google, Microsoft, and Anthropic said they will adopt a new standard that lets publishers express their opt-out preferences to multiple AI companies at once. The shared policy token signal was proposed at the IETF by Kevin Kelley, Anthropic's standards technical lead, and will be written into websites' robots.txt files; currently, websites must set preferences separately for each AI company. Anthropic has already adopted the signal, Google and Microsoft plan to support it next year, and OpenAI says it will implement it within the next few months; Microsoft will also offer more granular controls, allowing publishers to opt out of AI training or AI grounding separately, while their sites will still appear in Bing search results.

   Expand summary
7. 02:27

   Unite.AI

   OpenAI announces it will honor the shared robots.txt signal to express AI training preferences

   On October 30, 2026, OpenAI announced that it will honor the shared signal in robots.txt, allowing website operators to express the same AI training preference to multiple participating vendors at once, with implementation planned over the coming months. The signal is defined in an Internet-Draft submitted by Anthropic's Kevin Kelley; it uses the __usage_ai-training policy token bound to paths, adds no new directive names or syntax, and follows the parsing rules of RFC 9309. OpenAI stated that existing controls such as GPTBot and OAI-SearchBot remain in effect, and encouraged other AI vendors to honor the signal as well.

   Expand summary