Google’s latest documentation updates for the Google-Extended crawler bring practical changes that affect website owners and content managers. These modifications focus on AI training permissions and search visibility, with specific updates reflecting the transition from Bard to Gemini Apps. The changes give site owners direct control over how their content supports AI development.
A Little More About Google-Extended
Since its launch on September 28, 2023, Google-Extended has been offering web publishers a new way to manage their site’s visibility. It’s all about control. You can tell Google-Extended to take a hike or welcome it in, depending on whether you want your site’s content to train AI models. This choice is made possible through the Robots Exclusion Protocol, where you can set the rules of engagement for this particular crawler.
The technical implementation of Google-Extended builds upon established web crawling standards. Website administrators use specific commands in their robots.txt files to set permissions. These commands act as digital barriers or welcome signs, determining whether AI training systems can access and process the site’s content.
Though Google calls it a “standalone product token,” that term might sound like jargon. Simply put, it’s a way for you to specify how Google’s AI-focused crawlers interact with your site.
The initial announcement made things clear:
“We’re rolling out Google-Extended to give you more control. It’s your call if you want your site to contribute to making Bard and Vertex AI’s generative APIs smarter. This decision could influence how advanced these AI models get over time.”
Website owners who want to restrict access can implement a simple code block in their robots.txt file:
makefile
User-agent: Google-Extended
Disallow: /
Keeping Up with Changes
Google’s good at keeping a log of updates and changes, especially ones that matter to web publishers and marketing consultants. They’ve tweaked the Google-Extended documentation, especially after rebranding Bard to Gemini Apps. Now, Google-Extended’s crawling efforts are aimed at Gemini Apps and Vertex AI, but here’s the kicker: it won’t mess with your Google Search standing.
The separation between AI training and search indexing represents a significant technical distinction. Search crawlers continue their regular operation, maintaining the existing SEO structures and ranking factors. Meanwhile, Google-Extended operates as an independent system, collecting data specifically for AI model training.
So, What's the Big Update?
The main takeaway from the recent changes is that Google-Extended’s crawling is now focused on Gemini Apps, and it leaves Google Search rankings untouched.
In their own words:
“We’ve updated our terms to reflect Bard’s name change to Gemini Apps. Based on your feedback, we’ve made it clear: Google-Extended’s crawling is all about Gemini Apps and doesn’t affect Google Search.”
They’ve ditched the Bard name in favor of Gemini and added a reassuring note:
“Google-Extended won’t impact how your site ranks or appears in Google Search.”
Technical Implementation Details
The robots.txt configuration accepts several variations to give site owners precise control. Administrators can block specific directories while allowing others, creating a balanced approach to content sharing. For example:
User-agent: Google-Extended
Allow: /public/
Disallow: /private/
This granular control helps organizations maintain a strategic balance between contributing to AI advancement and protecting sensitive information.
Practical Applications and Benefits
Website owners now have three clear options for managing their content’s role in AI development:
1. Full access: Allow Google-Extended to crawl all content, supporting AI model training across their entire site.
2. Partial access: Use directory-specific rules to share selected content while protecting other areas.
3. Complete restriction: Block Google-Extended entirely, opting out of AI training contributions.
These choices let organizations align their content strategy with their business objectives and data-sharing preferences.
Future Implications
The separation between search rankings and AI training creates new opportunities for content strategy. Organizations can now participate in AI development without concerns about search visibility impact. This change opens doors for selective content sharing based on business goals rather than SEO considerations.
For those managing a website and looking to fine-tune their exposure to AI training without affecting search visibility, these updates provide clarity and control. PushLeads highlights these changes to ensure you’re in the know, especially if you’re navigating the complexities of online visibility and search engine optimization as a marketing consultant.
Read Next: Essential Principles of Keyword Research
CLICK HERE to schedule your FREE consultation TODAY!
