Four AI Companies Agreed on One Opt-Out Line for Websites. Not One of Them Reads It Yet, and They Will Not Read It the Same Way

Google, Microsoft and OpenAI said Friday that they plan to honor a shared line in websites' robots.txt files that would let a site state one AI-training preference to every participating provider, instead of writing separate rules for each company's crawler, Search Engine Journal reported. Anthropic, whose engineer proposed the mechanism, has also signed on, according to Nieman Lab.

For publishers, the promise is simplicity. Today, a site that does not want its content scraped for AI training must block each company's crawlers individually, Nieman Lab noted. OpenAI said the shared signal 'makes it easier for publishers and website operators to communicate the same preference to multiple participating providers at once.'

<h3>Not a standard, and not yet in force</h3>

The line comes from an Internet Draft that Anthropic's Kevin Kelley submitted to the AI Preferences working group of the Internet Engineering Task Force. The IETF's document tracker lists it as an individual draft, which does not represent official IETF standardization, Search Engine Journal reported. The draft expires May 3, 2027, unless replaced, and Kelley plans to propose his design as a change to another working-group draft, according to SEJ.

Based on the companies' own timelines, none of the three currently supports the line, SEJ reported. Microsoft says it will begin honoring new preferences on Feb. 1, 2027. Google's crawler documentation says early 2027. OpenAI says it aims to honor the signal 'in the next few months,' while its current robots.txt policy remains in effect. Nieman Lab reported that Anthropic is already supporting it, while SEJ reported that Kelley wrote on the working group's mailing list that implementing support across all of Anthropic's services will take time.

<h3>How the line works</h3>

Kelley's draft adds no new directive. A site writes an ordinary rule group under a 'policy token,' __usage_ai-training, using familiar Allow and Disallow rules. The draft's example opts an entire site out of training except a /press-kit/ folder, SEJ reported.

The token's meaning comes from the working group's vocabulary draft, which defines AI training as using content to modify a generative AI model's learned parameters; models used only for non-generative tasks such as classification or ranking are exempt.

Two details matter for site owners. Existing robots.txt rules will not count as a shared training preference, and a wildcard group (User-Agent: *) can never match the token. A file without a __usage_ai-training group states no preference, which Kelley says is true of every robots.txt file currently published. The token also does not control crawling: crawlers may still fetch pages, SEJ reported.

<h3>Same line, different results</h3>

Google said that when a site opts out of training with the shared line and does not specify Google-Extended, it will treat that 'as if the publisher had disallowed Google-Extended,' according to a post by Google's John Mueller cited by SEJ. Google-Extended controls both training of future Gemini models and grounding in Gemini Apps and on Vertex AI. That means, as SEJ noted, Google's documentation describes no way to keep content out of Gemini training while still allowing Gemini Apps to use it for grounding. Google-Extended does not affect inclusion in Google Search.

Microsoft is splitting the two. Its post, by Bing principal product manager Krishna Madhavan, adopts the shared line for training and adds a separate page-level AI Grounding preference, with an option called DisallowExtended that allows grounding in Microsoft's own tools but not through grounding services it sells to other businesses. Microsoft will stop recognizing its older NoArchive and NoCache tags as training or grounding controls after March 1, 2027, and sites using those tags must record a temporary preference in Bing Webmaster Tools, available from Dec. 15, to stay out of grounding in the meantime, SEJ reported.

Neither OpenAI's post nor its crawler documentation, as checked Oct. 30, says what happens when a site's shared-line rule conflicts with its GPTBot rule, SEJ reported.

<h3>Who else</h3>

Meta's vice president of communications, Andy Stone, called the signal 'an important step' in a post, saying the company supports 'publishers' ability to express preferences about how their content is used for foundational generative AI training,' Nieman Lab reported. Meta did not, in that report, commit to a date. Meta is under scrutiny over proposed commercial terms that would allow broad scraping and agreed last week to delay them, according to Nieman Lab.

<h3>What remains unknown</h3>

The announcements do not address content already used for training, how a publisher could verify that a preference was honored, or whether the IETF will adopt the draft as a standard. Until the companies' start dates arrive, the shared line is a commitment, not a control.