Google and Microsoft to honor a shared AI opt-out in 2027
One robots.txt rule will let a site refuse AI training, but the signal is an IETF draft and no one reads it yet.
A website will be able to refuse the use of its content for AI training with a single group of rules in its robots.txt file, the file that tells crawlers which pages they may visit. Google and Microsoft said on Friday they will honor that signal, but neither reads it today.
Google says it will support the control in 2027, along with other AI providers. It will sit alongside Google-Extended, the company's own control since 2023: if a site uses the shared signal and does not mention Google-Extended, Google will treat it as if the site had blocked Google-Extended.
Microsoft was more specific. It will honor the new preferences when it crawls starting Feb. 1, 2027, and it splits the choice in two. Training uses the shared signal and can be fine-tuned page by page with a meta tag or an HTTP header. Grounding, meaning the use of content to build AI answers, can only be set page by page, with three values:
- Allow: the default.
- DisallowExtended: limits use to Microsoft's own experiences, such as Copilot, and excludes third-party services.
- Disallow: the content is not used to generate answers.
Blocking grounding cuts a site's presence in Microsoft's AI experiences. March 1, 2027, is the deadline: after that, NoArchive and NoCache stop counting as a preference for training or grounding.
The signal comes from an IETF draft titled "Machine-Readable Training Preference Signal," signed by Kevin Kelley of Anthropic and dated Oct. 30. It defines the token `__usage_ai-training`, written on the User-Agent line as if it were one more crawler. A group with `Disallow: /` and `Allow: /press-kit/` refuses training across the whole site except the press section.
Its reach is narrow. The token states a preference about use and does not block access, and the text offers no way to force anyone to comply. A `User-agent: *` group does not count for it, so a robots.txt that shuts out every crawler says nothing about training. And it is an individual draft, not endorsed by the IETF, that expires May 3, 2027.
For a Mexican site, the practical change arrives next year. Until then, per-crawler blocks such as Google-Extended are what each company reads today, and Microsoft wants preferences published before March 1.
