One Line in robots.txt to Say "No AI Training"? OpenAI Says It Will Honor a Shared Signal
Sources: OpenAI's official announcement (Oct 30, 2026), the IETF Internet-Draft draft-kelley-aipref-training-preference-signal-00, and OpenAI's crawler documentation
Oct 30, 2026: Website owners who want to keep their content out of generative AI training have usually had to block one crawler at a time. OpenAI says that should get simpler. In a post published Oct 30, 2026 (the RSS feed timestamp is 09:30 GMT, about 5:30 AM ET), the company said it "will honor a common signal in robots.txt" that lets sites state their AI training preference to multiple participating providers at once.
The catch: it isn't live yet. OpenAI says it aims to implement the signal "in the coming months."
What OpenAI announced
According to OpenAI's post:
- OpenAI will honor a shared robots.txt signal for preferences about using content in "foundational generative AI training."
- Before this, operators "would identify individual crawlers and maintain separate rules for each and every provider." OpenAI says a shared signal "reduces that work for operators."
- It "will not replace" OpenAI's existing robots.txt controls, which "will continue to apply and be respected."
- The work draws on the IETF AI Preferences (aipref) working group. OpenAI says it is "joining other participants in the working group in today's announcement."
- OpenAI "encourage[s] other AI providers to honor the signal."
How the signal works (per the IETF draft)
OpenAI links to an Internet-Draft titled "Machine-Readable Training Preference Signal", dated Oct 30, 2026. The listed author is Kevin Kelley, with the affiliation Anthropic. Here's what the draft says:
- It defines one cross-vendor policy token:
__usage_ai-training. - You use it like a regular robots.txt user-agent group. The draft's own example:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
- According to the draft, that example states a preference against AI training everywhere on the site except
/press-kit/, where it states a preference in favor. - It adds no new directives or grammar. Parsers that don't support it simply ignore the group.
- A wildcard group (
User-Agent: *) must not be treated as matching the token. A blanketDisallowfor all bots does not count as an AI training preference under this mechanism. - If no group names
__usage_ai-training, no preference has been expressed. - The token is "not a directive for crawling behavior." If you want to limit crawling, the draft says to keep using crawler-specific groups.
The draft defines "AI Training" as "Using an asset to modify the learned parameters of a generative AI model."
Important caveat: the IETF datatracker says this individual draft "is not endorsed by the IETF and has no formal standing in the IETF standards process." It's a proposal, not a finished standard.
What doesn't change: GPTBot and friends
OpenAI's crawler docs still list separate tokens. Disallowing GPTBot "indicates a site's content should not be used in training." OAI-SearchBot controls whether a site shows up in ChatGPT search answers. Per the announcement, these controls stay in place alongside the new signal.
Why it matters for site owners
If other AI companies adopt the same token, one robots.txt group could replace a growing list of vendor-specific rules. It also separates may you crawl me? from may you train on what you crawl? It's still a preference signal, not access control: it relies on providers choosing to honor it.
What's confirmed / not confirmed
Confirmed (OpenAI, Oct 30, 2026): - OpenAI will honor a common robots.txt signal for AI training preferences. - Implementation is targeted for "the coming months," with a follow-up update promised. - Existing OpenAI robots.txt controls stay in effect.
Confirmed (IETF draft text):
- The token is __usage_ai-training. Wildcard groups don't match it, and it doesn't govern crawling.
- The draft is an individual Internet-Draft with no formal IETF standing. It expires May 3, 2027.
Not confirmed: - An exact date when OpenAI will start honoring the token. - Which other providers have committed. OpenAI doesn't name them, and third-party reports of other companies' timelines weren't verified, so we've left them out. - Whether it affects content collected before a site adds the token.
Sources
- OpenAI, "Making it easier for websites to express AI training preferences" (Oct 30, 2026): https://openai.com/index/helping-websites-express-ai-training-preferences/
- IETF datatracker, draft-kelley-aipref-training-preference-signal-00: https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/
- OpenAI crawler overview: https://developers.openai.com/api/docs/bots
Related
Up: AI
