OpenAI will honor a shared robots.txt signal for opting out of AI training
OpenAI says it will start honoring a common signal in robots.txt, the file websites use to set rules for crawlers, so site operators can state their preference about foundational generative AI training once instead of...
·1 min read·AB
OpenAI says it will start honoring a common signal in robots.txt, the file websites use to set rules for crawlers, so site operators can state their preference about foundational generative AI training once instead of writing separate rules for every AI company. The company describes it as a voluntary effort that lets publishers send the same preference to multiple participating providers at once.
OpenAI says it plans to implement the signal in the coming months and will share more details when it does. The new signal does not replace OpenAI's existing robots.txt crawler controls, which will keep applying as before.
The effort builds on work by the IETF's AI Preferences working group, and OpenAI says it is joining other participants in that group in announcing the shared signal. OpenAI is encouraging other AI providers to honor it too.
Why it matters
A single opt-out signal would save website operators from maintaining separate AI-crawler rules for every provider.
Source: OpenAI