OpenAI, Google, Microsoft, and Anthropic will adopt a unified AI crawling opt-out standard
OpenAI, Google, Microsoft, and Anthropic said they will adopt a new standard that lets publishers express their opt-out preferences to multiple AI companies at once. The shared policy token signal was proposed at the IETF by Kevin Kelley, Anthropic's standards technical lead, and will be written into websites' robots.txt files; currently, websites must set preferences separately for each AI company. Anthropic has already adopted the signal, Google and Microsoft plan to support it next year, and OpenAI says it will implement it within the next few months; Microsoft will also offer more granular controls, allowing publishers to opt out of AI training or AI grounding separately, while their sites will still appear in Bing search results.
Why read this
AI score 62
Four major AI companies have agreed to adopt the same robots.txt signal, allowing readers to understand the mechanism by which publishers uniformly opt out of AI crawling and the differences in each company's implementation timeline.
Topics
Body
OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI, as reported earlier by Search Engine Roundtable . With this control, publishers can apply their AI training preferences across multiple AI companies, instead of setting them individually.
The shared “policy token” signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF). It will live within a website’s robots.txt file , which has long signaled which web crawlers can and can’t scrape a site — whether for search indexing, archiving, or AI training. Currently, sites have to signal their AI training preferences separately to each AI company.
The announcement comes as publishers grapple with the continuing influx of AI crawlers scraping their sites and feeding their content into AI models. Some companies and organizations have attempted to combat unauthorized scraping, with the Really Simple Licensing (RSL) standard giving publishers the ability to charge crawlers that visit their sites, and Cloudflare blocking AI crawlers by default .
OpenAI , Google , Microsoft , and Anthropic say they will restrict their AI crawlers from accessing a site if they encounter the shared token. Microsoft, however, will offer more granular controls that will give publishers the ability to opt out of AI training or AI grounding , the process of pointing an AI model to a verified source of information to reduce hallucinations. Microsoft notes that websites will still appear in Bing’s search results regardless of their preferences.
While Anthropic has already adopted the signal, Google and Microsoft plan to support it next year. OpenAI says it will implement it in the “coming months.”