AI News

Google, OpenAI, Microsoft and Anthropic Back One AI Training Opt-Out for Websites

Future AI Wire Editorial Team · November 2, 2026 · 6 min read
AI training opt-out: a website page and shield blocking data streams from reaching an AI neural core

A single AI training opt-out for websites is on the way: Google, OpenAI, Microsoft and Anthropic said on October 30, 2026 that they will support one shared robots.txt signal that tells AI developers a site does not want its content used to train models, according to The Next Web and Coywolf.

This article reflects reporting available as of November 2, 2026.

How the AI Training Opt-Out Works

The proposal is an IETF Internet-Draft, draft-kelley-aipref-training-preference-signal, written by Kevin Kelley of Anthropic. It adds a new “policy token” to robots.txt called __usage_ai-training. Instead of listing each company’s crawler separately, a site owner adds one group for that token and lists the paths that should not be used for training.

The draft’s example blocks training on a whole site except a press kit folder, as reported by Coywolf:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

According to The Next Web, the draft describes the token as a statement of preference about how content is used, not an instruction about crawling. A site with no such group has expressed no preference, and existing rules aimed at all crawlers do not count as an opt-out.

When Each Company Says It Will Honor the Signal

  • Google: plans support in early 2027. If a site uses the new token but does not mention Google-Extended, Google will treat it as disallowing Google-Extended, according to The Next Web.
  • Microsoft: will honor it from February 2027. Microsoft also announced separate controls for “grounding,” when an AI answer such as Copilot draws on a live page. Microsoft said the change does not affect where a page ranks in Bing search.
  • OpenAI: said it will implement the signal “in the coming months.” Its existing controls, including GPTBot rules, still apply.
  • Anthropic: has started work. Kelley said it will take time to add support across Anthropic’s services and handle edge cases.

Reports differ slightly on Anthropic’s status. Coywolf wrote that Anthropic is the only company using the token today, while The Next Web quoted Anthropic saying full support will take time.

What the AI Training Opt-Out Does Not Do

  • It is voluntary. There is no enforcement or independent check. Bots that ignore robots.txt will still ignore it, Coywolf noted.
  • It covers training, not every AI use. TechBuzz pointed out it does not settle how AI products use web content in AI-generated answers.
  • It creates no payment deal. Microsoft said the new preferences do not set up licensing or payment arrangements.
  • It is not yet an IETF standard. The draft is an individual submission with no formal standing. A separate IETF working-group draft from Google’s Gary Illyes and Mozilla’s Martin Thomson takes a different approach using a Content-Usage rule.
  • Not every AI company is in. Meta’s communications head voiced support for a common signal but did not commit to honoring it, according to The Next Web. Smaller labs and open-source projects are not part of the agreement.

What Website Owners Should Do Now

Nothing changes overnight. Google and Microsoft will not honor the token until 2027, and OpenAI’s timeline is open. Until then, keep any existing rules for crawlers such as GPTBot, Google-Extended and ClaudeBot in place. This matters more as AI agents increasingly browse and read websites on users’ behalf. All four companies say their current controls remain valid.

Publishers should also treat training and AI search visibility as two separate decisions. Microsoft warned that blocking grounding will significantly reduce a site’s visibility in AI-generated experiences, so blocking everything may cost traffic from AI answers.

Training vs. Grounding vs. Search: Three Separate Choices

The announcements make clear that “AI use” is no longer one switch. Publishers now face three distinct decisions:

  • Training: Whether your pages can be used to build or improve AI models. This is what the new __usage_ai-training token covers.
  • Grounding: Whether an AI assistant can pull from your live page to answer a user’s question, often with a link back. Microsoft is adding separate controls for this.
  • Search indexing: Whether your pages appear in regular search results. The new training preference does not change this, according to Microsoft.

How to Prepare Your robots.txt for the AI Training Opt-Out

  1. Audit your current file. Note any existing rules for GPTBot, Google-Extended, ClaudeBot and other AI crawlers.
  2. Decide your training policy. Choose whether to block training site-wide, block only certain folders, or allow it.
  3. Add the new group alongside existing rules. Do not remove crawler-specific rules yet, because the companies will not honor the new token until their rollouts are complete.
  4. Test the file. Use a robots.txt testing tool, such as the robots.txt report in Google Search Console, to confirm normal search crawling is not blocked by mistake.
  5. Revisit in early 2027. Check Google’s and Microsoft’s documentation as their support goes live.

Why the AI Training Opt-Out Matters

Before this AI training opt-out, site owners had to track a growing list of crawler names to keep content out of AI training. A shared token is simpler, but it only works if the companies follow through and others join. The next milestones are Google’s and Microsoft’s 2027 rollouts and whether the IETF adopts a single approach.

Frequently Asked Questions

What is the AI training opt-out token?

It is a new robots.txt policy token, __usage_ai-training, proposed in an IETF draft. A website lists paths under it to signal that those pages should not be used to train AI models.

Does the AI training opt-out stop AI crawlers from visiting my site?

No. It states a preference about training use, not a crawl ban. Bots that ignore robots.txt will also ignore it, so server-level bot blocking is still needed for that.

When will Google and Microsoft honor the new token?

Google plans support in early 2027, and Microsoft says it will honor the signal from February 2027. OpenAI said “in the coming months,” and Anthropic said its rollout will take time.

Will blocking AI training hurt my Google or Bing rankings?

Microsoft said the training preference does not affect Bing search rankings. Separately blocking AI grounding can reduce visibility in AI-generated answers, so publishers should weigh training and AI answer visibility as separate choices.

Sources