Google has announced support for a unified mechanism in robots.txt that will allow website owners to opt out of having their content used to train generative AI models. The change is intended to simplify the process for small and medium-sized businesses as well as larger publishers who currently must manage separate instructions for each AI provider.
The company detailed the plans in a post on its Search Central blog. The new capability will be based on work from the AIPREF working group at IETF and is expected to become available in 2027. A common policy marker will be added to robots.txt so that sites can express a single preference regarding AI training instead of maintaining individual directives for every participating company.
Google stated that it will interpret a general prohibition expressed through the new marker in the same manner as the existing Google-Extended directive when no specific instruction for Google-Extended is present. At the same time, current per-provider settings will continue to function, enabling publishers to set a broad rule and then add targeted exceptions for selected AI companies.
For content owners the approach could reduce the administrative burden created by the growing number of AI crawlers. However, the company emphasized that robots.txt only signals usage preferences and does not prevent access to servers. The practical impact will therefore depend on how many organizations adopt the standard and consistently respect the instructions it contains.
The initiative remains a proposal that will continue to be refined and standardized through the IETF process before final implementation.
