🔍 Read the full analysis: What To Know Before Setting The AI Training Opt-Out on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI, Google, Microsoft and Anthropic have said they will honor a shared robots.txt signal asking that a website’s content not be used to train generative AI models. The signal is voluntary, is being introduced on different schedules and does not block crawling, affect AI answers or remove material from models already trained.
OpenAI, Google, Microsoft and Anthropic say they will honor a shared robots.txt signal that asks them not to use a website’s content to train generative AI models. The proposal could spare publishers from maintaining separate training opt-outs for each participating company, but it is a voluntary request—not a technical block—and the companies plan to adopt it on different schedules.
The shared rule is described in an individual Internet-Draft titled “Machine-Readable Training Preference Signal.” It uses the user-agent name __usage_ai-training with a disallow rule for the site. A publisher can add the group to its robots.txt file to express a preference against training use. The draft was published on October 30 by Kevin Kelley of Anthropic.
The companies’ commitments and timelines differ. Anthropic says it has started implementing support, while warning that support across its services will take time. OpenAI says it aims to honor the signal “in the coming months.” Google’s crawler documentation says it aims to support the control in early 2027. Microsoft says it will honor the training preference beginning February 1, 2027, as it crawls websites.
Microsoft also announced a separate page-level preference for AI Grounding, which concerns use of web content to inform answers in Bing, Copilot and Microsoft 365. Microsoft says the new preferences replace NoArchive and NoCache for training or grounding purposes after March 1, 2027. Meta’s communications chief voiced support for publishers expressing preferences, but did not commit the company to honoring this signal.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
For publishers, the change could make it easier to communicate a training preference to several major AI companies through one rule rather than maintaining a separate instruction for every crawler. That matters because a missing or outdated company-specific rule may not communicate a publisher’s preference to that company. The practical benefit, however, depends on which services adopt the signal and when.
The distinction between a request and a block is central. Robots.txt communicates instructions to crawlers; it does not technically prevent a company from accessing a page or guarantee compliance. The proposal also addresses model training only. It does not, by itself, stop a service from using live web content to generate answers, promise payment, or ensure a publisher’s material will be excluded from models that have already been trained.
Microsoft says its new training and grounding preferences do not guarantee inclusion in search results, traffic or payment. It also says the preferences do not change eligibility for Bing search results. Google says Google-Extended does not affect Google Search. These distinctions mean site owners may need to make separate decisions about training, AI-generated answers and search, rather than treating one opt-out as a universal switch.
Products Worth Considering
LG 34GX90SA-W 34-inch Ultragear WQHD OLED Curved Gaming Monitor
LG UltraGear 27G640A-B 27-Inch QHD IPS Gaming Monitor, 300Hz, 1ms, G-SYNC Compatible, FreeSync Premium, DisplayHDR 400
LG UltraGear 27-Inch QHD OLED Gaming Monitor, 240Hz, 0.03ms, G-SYNC Compatible, FreeSync Premium Pro, HDMI 2.1
As an affiliate, we earn on qualifying purchases.
How the Proposal Took Shape
The proposal is connected to the AI Preferences working group at the Internet Engineering Task Force, which develops internet standards. The shared signal is an individual Internet-Draft, not an adopted IETF standard. At a working-group meeting on October 21, one of the group’s chairs said, “We haven’t declared consensus on anything yet.” The draft’s publication and company announcements therefore do not establish formal standards-body approval.
Before the proposal, publishers generally had to use company-specific crawler rules to state preferences. Examples include GPTBot for OpenAI, Google-Extended for Google and ClaudeBot for Anthropic. Microsoft’s older NoArchive and NoCache tags have also been used for relevant preferences. The companies say those existing controls continue to apply during the rollout; site owners should not assume every service has switched to the shared signal.
The announcements point to the same draft, but they do not amount to a commitment from every AI company. Meta has expressed support for the idea without confirming implementation. The source material reviewed reports no statement on adoption from Apple, Amazon, xAI or Perplexity. Their positions should not be inferred from the draft’s acknowledgments or from participation by people associated with other organizations.
““has already started implementing support for this draft””
— Anthropic
Adoption and Enforcement Remain Open
Several details are unsettled. The signal is not an official internet standard, and the draft may change as the working group considers it. The companies have announced different implementation windows, and some have not specified exactly which products or services will recognize the rule. Anthropic’s crawler help page, as of October 31, still described its existing crawler controls rather than the shared signal.
It is also unclear whether Meta will honor the signal or when any additional AI companies might adopt it. A website’s robots.txt rule expresses a preference but does not provide technical enforcement, and the material does not establish how compliance will be monitored or what recourse a publisher would have if a company ignored it. The signal’s effect on specific uses beyond the stated training scope also should not be assumed.
Watch the Rollout Dates
Publishers can review their existing crawler rules and decide whether to add the shared training preference, while keeping company-specific instructions in place until each service confirms support. Sites using Microsoft’s NoArchive or NoCache preferences should review Microsoft’s stated March 1, 2027 changeover and its separate AI Grounding control. Those steps communicate preferences; they do not guarantee that a company will comply or that content will be excluded from every AI-related use.
The next milestones are the companies’ implementation updates and the IETF working group’s consideration of the draft. Google targets early 2027, Microsoft’s stated start date is February 1, 2027, and OpenAI has said “in the coming months.” Until the details and adoption status are clearer, site owners should check the companies’ current crawler documentation rather than treating the shared rule as already universal.
Key Questions
What does the shared AI training opt-out do?
It is a robots.txt preference asking participating companies not to use a website’s content to train generative AI models. It does not technically block access to the site.
Which companies have said they will honor it?
OpenAI, Google, Microsoft and Anthropic have announced support, with different implementation timelines. Meta has expressed support for publishers’ ability to set preferences but has not confirmed that it will honor this signal.
Does the rule stop AI services from using pages in answers?
Not by itself. The shared signal described in the draft concerns model training. Microsoft separately announced an AI Grounding preference for certain products; that is distinct from the shared training rule.
Should website owners remove their existing crawler rules?
The announcements say existing company-specific controls continue to apply. Site owners should check current documentation and rollout dates before removing those rules. Microsoft says NoArchive and NoCache stop counting as training or grounding preferences after March 1, 2027.
Is this the setting for opting out of AI training on my chats?
No. This proposal concerns website content and robots.txt, not conversations in an individual AI account. Chat-data preferences are handled through the relevant service’s account settings.
Source: ThorstenMeyerAI.com
