Google-Extended

Google to Back Unified robots.txt Standard for Blocking AI Training Data Use Starting in 2027

Google has announced plans to support a common mechanism in robots.txt that lets website owners set a single preference against the use of their content for training generative AI models. The feature will rely on work from the AIPREF working group at IETF and is scheduled for rollout in 2027. Under the proposal, a general policy marker will be introduced so that sites can express preferences regarding AI training without issuing separate directives to each provider. Google intends to treat a general prohibition the same way it currently handles the Google-Extended directive when no specific instruction exists. Existing per-company settings will remain available, allowing publishers to combine a broad rule with targeted exceptions for individual AI providers. The company noted that robots.txt only communicates usage preferences and does not technically block crawler access, so effectiveness will depend on which organizations adopt and honor the new standard.