OpenAI、Google、Microsoft、Anthropic 将采用统一的 AI 抓取退出标准
The biggest names in AI are backing a new way to opt out of scraping
OpenAI、Google、Microsoft 和 Anthropic 表示将采用一项新标准，让出版商可以一次性向多家 AI 公司表达退出抓取的偏好。该共享 policy token 信号由 Anthropic 标准技术负责人 Kevin Kelley 在 IETF 提出，将写入网站的 robots.txt 文件，目前网站需分别向各家 AI 公司设置偏好。Anthropic 已采用该信号，Google 和 Microsoft 计划明年支持，OpenAI 称将在未来几个月内实施；Microsoft 还将提供更细粒度控制，允许出版商分别退出 AI 训练或 AI grounding，但网站仍会出现在 Bing 搜索结果中。
推荐理由
AI 评分 62
四家主要 AI 公司同意采用同一 robots.txt 信号，读者可了解出版商统一退出 AI 抓取的机制与各家落地时间差。
主题
正文
OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI, as reported earlier by Search Engine Roundtable . With this control, publishers can apply their AI training preferences across multiple AI companies, instead of setting them individually.
The shared “policy token” signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF). It will live within a website’s robots.txt file , which has long signaled which web crawlers can and can’t scrape a site — whether for search indexing, archiving, or AI training. Currently, sites have to signal their AI training preferences separately to each AI company.
The announcement comes as publishers grapple with the continuing influx of AI crawlers scraping their sites and feeding their content into AI models. Some companies and organizations have attempted to combat unauthorized scraping, with the Really Simple Licensing (RSL) standard giving publishers the ability to charge crawlers that visit their sites, and Cloudflare blocking AI crawlers by default .
OpenAI , Google , Microsoft , and Anthropic say they will restrict their AI crawlers from accessing a site if they encounter the shared token. Microsoft, however, will offer more granular controls that will give publishers the ability to opt out of AI training or AI grounding , the process of pointing an AI model to a verified source of information to reduce hallucinations. Microsoft notes that websites will still appear in Bing’s search results regardless of their preferences.
While Anthropic has already adopted the signal, Google and Microsoft plan to support it next year. OpenAI says it will implement it in the “coming months.”