Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard — including opting-out of internet scraping

Google, Microsoft, OpenAI, and Anthropic say they will honor one robots.txt group as a cross-company opt-out from AI training It governs how content is used rather than whether bots can crawl it, defaults to allowing training, and does nothing for content already scraped Anthropic has started implementing it, OpenAI promises to do so in the coming months, Microsoft starts February 1, 2027, and Google targets 2027, while Meta is supportive but uncommitted Google and Microsoft have joined OpenAI and Anthropic in promising to honor a single line in robots.txt that tells AI developers whether a website's content can be used to train their models.