Google recently made changes to its Google-Extended web crawler user agent documentation, addressing the impact on search and providing clearer guidance on controlling content access for AI model training. The update may worry those who prefer to block the crawler.
The Google-Extended user agent was introduced on September 28, 2023, and gives web publishers the option to control how their sites are crawled. By using the Robots Exclusion Protocol, publishers can either allow or disallow the Google-Extended user agent, enabling them to opt-out of having their content scraped and included in AI training datasets.
Google clarified that Google-Extended is a “standalone product token,” which may be confusing to publishers familiar with the concept of user agents. The original announcement describes the user agent as a control for web publishers to manage whether their sites contribute to improving Bard and Vertex AI generative APIs.
Blocking Google-Extended is achieved using the “Google-Extended” User Agent, and Google maintains a changelog of important updates made to its guidance and communication with web publishers and the search marketing community. The revision to the Google-Extended documentation came after the renaming of Bard to Gemini Apps, specifying that Google-Extended’s indexing now contributes to Gemini Apps and Vertex AI generative APIs.
The updated guidance assures publishers that opting out of Google-Extended AI data collection does not affect Google Search, as Google-Extended crawling is exclusive to Gemini Apps and has no impact on Google Search. The changelog advises that Bard has been replaced with Gemini and emphasizes that Google-Extended does not impact a site’s inclusion or ranking in Google Search.
For more information, please refer to Google’s updated crawler overview, “Overview of Google crawlers and fetchers (user agents).”
GIPHY App Key not set. Please check settings