Google-Extended robots.txt: Official Guide for Sites

Jeremy Ashburn Jeremy Ashburn · Founder, PushLeads 14 min read

The short answer

Google-Extended is a standalone robots.txt token that controls whether Google can use your content to train Gemini and related AI products. Blocking it prevents that training use but has no effect on normal Google Search indexing or your eligibility to appear in AI Overviews.

79%
Of top US/UK news sites block an AI training crawler
336%
Rise in AI crawler blocking in a year
73,000:1
Anthropic crawl-to-referral ratio, June 2025
14:1
Google Search crawl-to-referral ratio

Watch the video version

Key moments
0:00 Google-Extended: Control Your AI Crawler Access
0:26 Most Sites Are Open by Default
0:48 What Google-Extended Actually Is
1:24 Googlebot vs. Google-Extended
2:02 Your Robots.txt Is Deciding Right Now
2:28 How to Block Google-Extended Step by Step
3:15 The Crawl-to-Referral Problem
3:46 AI Crawler Blocking Across Top Domains
4:24 Why Publishers Are Pushing Back
4:50 Publisher Blocking Is Accelerating Fast
5:21 Local Businesses Have Real Content at Stake
6:01 Other AI Crawlers Hitting Your Site Now
6:37 Google-Extended and AI Overviews: The Real Answer
7:11 How to Verify Your Robots.txt Is Working
7:58 Full AI Crawler Reference at a Glance
8:38 Take Control of Who Crawls Your Site
Full transcript

0:00 Google-Extended: Control Your AI Crawler Access
Your robots.txt file is making decisions about AI data access right now — whether you wrote those rules intentionally or not. We're walking through exactly what Google-Extended is, how it differs from the crawler controlling your search rankings, and how to make a deliberate choice about who trains AI models on your content. This is PushLeads, and this is your complete guide to Google-Extended.

0:26 Most Sites Are Open by Default
Only five point six percent of the top ten thousand domains have explicit rules for Google-Extended in their robots.txt files. Roughly ninety-four out of every one hundred websites have made no deliberate choice about this crawler. They're open by default. And default is not a strategy — it's just the absence of one.

0:48 What Google-Extended Actually Is
Google announced Google-Extended in September two thousand and twenty-three to give publishers a clean, documented way to separate AI training access from search crawling. Before that, there was no easy mechanism to say yes to search indexing and no to AI data collection simultaneously. Google-Extended reads your published pages and sends that content to Google to train Gemini and future AI products. It's completely separate from Googlebot, uses a different user agent token, and follows its own robots.txt rules. It's an officially documented, fully supported feature.

1:24 Googlebot vs. Google-Extended
Googlebot and Google-Extended are two different crawlers with two completely different jobs. Googlebot crawls and indexes your site for Google Search — it affects your rankings and drives referral traffic. Blocking it removes you from search results entirely. Google-Extended does none of that. Its only job is collecting content for AI model training. It has zero effect on your search rankings, your Maps listing, or any other Google search product. Blocking it is completely safe. The two follow separate directives in your robots.txt, and blocking one has absolutely no effect on the other.

2:02 Your Robots.txt Is Deciding Right Now
Every website already has a robots.txt file talking to every crawler that visits — including Google-Extended. If you've never added an explicit rule for it, the file isn't staying neutral. Silence means open access. No rule equals full permission. The product token is simply Google-Extended. Two lines in your robots.txt puts you in complete control of that pipeline without touching anything related to search.

2:28 How to Block Google-Extended Step by Step
Step one: find your robots.txt file at your domain slash robots dot txt — open it in any text editor. Step two: add the user agent block — type User-agent: Google-Extended on its own line. Step three: write the disallow directive — Disallow: / blocks the entire site, or use specific paths like slash blog slash to protect only certain sections. Step four: save and upload the file, replacing the existing version at your domain root. Step five: open Google Search Console, find the robots.txt report under Settings, and confirm the file reads exactly as you wrote it. Step six: use the URL inspection tool to verify Googlebot can still reach your pages.

3:15 The Crawl-to-Referral Problem
The crawl-to-referral ratio for AI training crawlers is staggering. Google Search runs at roughly fourteen to one — fourteen pages crawled for every visit returned through rankings. That's a reasonable exchange. Anthropic's crawler ran at seventy-three thousand to one in June two thousand and twenty-five, according to Cloudflare data. Seventy-three thousand pages consumed for every single referral visit returned to publishers. For any business depending on website traffic to generate leads, that math matters enormously.

3:46 AI Crawler Blocking Across Top Domains
The publisher response is measurable and growing fast. Cloudflare's analysis of the top ten thousand domains shows GPTBot is the most frequently blocked AI crawler, with three hundred and twelve of those domains blocking it. Google-Extended ranks third at five point six percent of analyzed robots.txt files. ClaudeBot is now blocked by approximately five point eight million websites — up from three point two million as recently as July two thousand and twenty-five. Publishers have figured out that targeted AI crawler control is real, effective, and carries no search ranking risk.

4:24 Why Publishers Are Pushing Back
Harry Clarkson-Bennett, SEO Director at The Telegraph, put it plainly: publishers are blocking AI bots using the robots.txt because there is almost no value exchange. Large language models are not designed to send referral traffic, and publishers still need traffic to survive. That logic applies equally to a local contractor, a boutique, a photographer, or any small business whose website content drives revenue.

4:50 Publisher Blocking Is Accelerating Fast
Seventy-nine percent of top US and UK news websites now block at least one AI training crawler. The overall blocking rate increased three hundred and thirty-six percent over the prior year, according to Tollbit's second quarter two thousand and twenty-five report. ClaudeBot alone crossed five point eight million blocked sites in a matter of months. This is not a fringe concern — it's a mainstream business decision publishers across every size and sector are actively making.

5:21 Local Businesses Have Real Content at Stake
A hand-written guide about local renovation costs, a detailed service page explaining your process, a portfolio of completed projects — that content took real expertise and real time to produce. AI training systems find exactly that kind of specific, locally grounded content valuable. They consume it, train on it, and send no traffic back to the business that created it. If your website generates leads, your content is a real business asset. Making a deliberate decision about who can train AI models on that asset is part of protecting your business — not just a technical detail.

6:01 Other AI Crawlers Hitting Your Site Now
Google-Extended is one crawler in a much larger ecosystem. GPTBot is OpenAI's training crawler and the most frequently blocked across the web. CCBot is Common Crawl's bot, used by multiple AI companies, and the second most blocked. ClaudeBot is Anthropic's crawler for Claude AI. Bytespider is ByteDance's crawler, associated with TikTok's parent company. Each uses its own user agent token, meaning each needs its own separate disallow block. A catch-all disallowing everything would block Googlebot too — removing your site from search entirely. Be specific with every token.

6:37 Google-Extended and AI Overviews: The Real Answer
One common question: does blocking Google-Extended remove you from AI Overviews? Probably not directly. AI Overviews appear to pull from content Googlebot has already indexed — Google-Extended feeds model training pipelines. Those are separate processes. Google has not publicly documented a direct link between blocking Google-Extended and exclusion from AI Overviews. The reliable path to AI-generated search features is well-structured content with schema markup, clear headings, and direct answers near the top of your pages — with Googlebot able to access everything.

7:11 How to Verify Your Robots.txt Is Working
Writing the robots.txt rules is step one. Confirming they work as expected is step two. Open Google Search Console, find the robots.txt report under Settings, and confirm the file reads exactly as you wrote it. Check for conflicting rules — a specific block wins over a catch-all. Use the URL inspection tool to confirm Googlebot can still reach your pages. To verify a visitor claiming to be Google-Extended, run a reverse DNS lookup — the hostname should end in googlebot.com or google.com — then run a forward DNS lookup to confirm it returns the original IP. If you're not confident your file is correct, a technical SEO audit shows exactly what every crawler sees.

7:58 Full AI Crawler Reference at a Glance
Every AI crawler and its exact token: Google-Extended for Google's AI training — safe to block, no ranking impact. GPTBot for OpenAI — most frequently blocked across the web. ClaudeBot for Anthropic — five point eight million sites block it. CCBot for Common Crawl — second most blocked overall. Bytespider for ByteDance — requires its own separate block. The golden rule: never use a catch-all User-agent asterisk disallow. That one line blocks everything including Googlebot and removes your site from search entirely. List each AI crawler with its own explicit block and keep your Googlebot rules completely separate.

8:38 Take Control of Who Crawls Your Site
Your robots.txt is making AI access decisions for your business right now — whether you intended it to or not. Google-Extended, GPTBot, ClaudeBot, and others visit sites across the web every single day. Most small business sites have no explicit rules for any of them, which means they're open by default. A clear robots.txt strategy lets you keep full Google Search visibility while deciding exactly which AI systems can train on the content you worked hard to create. That is a real business decision. See exactly where your visibility stands with a free teardown — find the link in the description, or call eight two eight, three four eight, seven six eight six.

▶ Watch it hereMore videos →

Side by side

GooglebotGoogle-Extended
ControlsSearch crawling and indexingUse of content to train Gemini
Blocking affects rankingsYesNo
Blocking affects AI Overviews eligibility—No

AI search

Is your business showing up in AI answers?

Run a free audit on any page in about 20 seconds.

Run my free audit

Google-Extended is a standalone robots.txt token that controls whether Google can use your website content to train its Gemini AI models and related products. Blocking it prevents that specific training data use but has no effect on normal Google Search indexing or your eligibility to appear in AI Overviews results.

An Introduction to Google-Extended

Google-Extended lets you control whether your website content trains Google's AI models while keeping your search rankings completely untouched. According to Cloudflare's 2025 Year in Review, Google-Extended is among the most frequently disallowed user agents in robots.txt files, reflecting growing publisher concerns about AI training data usage.

The distinction matters because this crawler operates independently from Googlebot. You can block Google-Extended entirely and maintain full visibility in Google Search, Maps, and other search products. For website owners weighing their options, understanding how this crawler works provides a strategic advantage in the evolving relationship between publishers and AI platforms.

An Introduction to Google Extended

What Google-Extended Actually Does

The numbers tell a clear story. BuzzStream research found that 79% of top news websites in the US and UK now block at least one AI training crawler. The blocking rate increased 336% over the past year, according to Tollbit's Q2 2025 report.

"Publishers are blocking AI bots using the robots.txt because there's almost no value exchange," explains Harry Clarkson-Bennett, SEO Director at The Telegraph. "LLMs are not designed to send referral traffic and publishers still need traffic to survive."

The crawl-to-referral ratio explains the frustration. Cloudflare data shows Anthropic's ratio was 73,000:1 in June 2025, meaning it crawled 73,000 pages for every single referral it sent back to publishers. Google Search's ratio was 14:1. AI systems consume massive amounts of content while returning almost nothing to the original creators.

Specific blocking statistics from Cloudflare's analysis of top 10,000 domains:

  • GPTBot: Most frequently blocked AI crawler with 312 domains disallowing access
  • CCBot: Second most blocked
  • Google-Extended: Third most blocked, but only 5.6% of robots.txt files include it
  • ClaudeBot: Now blocked by approximately 5.8 million websites, up from 3.2 million in July 2025

Why Publishers Are Blocking AI Crawlers

The numbers tell a clear story. BuzzStream research found that 79% of top news websites in the US and UK now block at least one AI training crawler. The blocking rate increased 336% over the past year, according to Tollbit's Q2 2025 report.

"Publishers are blocking AI bots using the robots.txt because there's almost no value exchange," explains Harry Clarkson-Bennett, SEO Director at The Telegraph. "LLMs are not designed to send referral traffic and publishers still need traffic to survive."

The crawl-to-referral ratio explains the frustration. Cloudflare data shows Anthropic's ratio was 73,000:1 in June 2025, meaning it crawled 73,000 pages for every single referral it sent back to publishers. Google Search's ratio was 14:1. AI systems consume massive amounts of content while returning almost nothing to the original creators.

Specific blocking statistics from Cloudflare's analysis of top 10,000 domains:

  • GPTBot: Most frequently blocked AI crawler with 312 domains disallowing access
  • CCBot: Second most blocked
  • Google-Extended: Third most blocked, but only 5.6% of robots.txt files include it
  • ClaudeBot: Now blocked by approximately 5.8 million websites, up from 3.2 million in July 2025

How to Block Google-Extended (Step by Step)

Implementing the block takes less than five minutes. Add these lines to your robots.txt file in your website's root directory:

User-agent: Google-Extended
Disallow: /

This tells Google's AI training crawler to stay away from your entire site. Your search visibility remains completely unaffected because Googlebot operates independently.

For partial access, where you want some content available for AI training but not others:

User-agent: Google-Extended
Allow: /blog/public/
Disallow: /blog/premium/
Disallow: /research/

This configuration lets Google-Extended access your public blog posts while protecting premium content and research materials.

Verifying Your Implementation

After adding the directives, verify they're working:

  1. Check your robots.txt is accessible at yourdomain.com/robots.txt
  2. Use Google's robots.txt tester in Search Console
  3. Monitor server logs for Google-Extended user agent strings

Important note: Google-Extended is a control token, not a traditional crawler. According to technical guides, it won't appear in your server logs the same way Googlebot does.

The Bigger Picture: AI Crawler Traffic in 2025

Infographic showing 2025 AI crawler traffic and blocking statistics
AI bots now account for a significant portion of web traffic, with blocking rates rising 336% year over year.

AI crawlers now represent a significant portion of web traffic. Cloudflare's analysis found that AI bots (excluding Googlebot) accounted for 4.2% of all HTML requests across their network in 2025. Googlebot alone handled 4.5%, more than all other AI bots combined.

The growth has been explosive:

  • GPTBot traffic increased 305% between May 2024 and May 2025
  • Total crawler traffic rose 18% in the same period
  • AI "user action" crawling increased more than 15 times in 2025

Cloudflare research found that only 14% of the top 10,000 domains have specific rules for AI crawlers in their robots.txt files. This leaves the vast majority of websites unprotected, with AI systems freely collecting their content for training.

"It is never too late to block," notes Anthony Katsur, CEO of IAB Tech Lab. "The LLMs will come back and they will recrawl content in order for that information to stay fresh and relevant and accurate."

Strategic Considerations for Your Website

Comparison infographic for blocking or allowing Google-Extended crawler
For local service businesses, the calculation differs from major publishers whose revenue depends on direct site traffic.

The decision to block AI crawlers depends on your business model and content strategy. Consider these factors:

Block Google-Extended if you:

  • Publish premium or subscription content
  • Want to protect intellectual property
  • Generate revenue primarily through direct site traffic
  • Create original research or proprietary data

Allow Google-Extended if you:

  • Want visibility in AI-powered search results
  • Benefit from brand mentions in AI responses
  • Prioritize reach over content protection
  • Create educational or public-interest content

For local service businesses, the calculation differs from major publishers. Your Google Business Profile and local search rankings depend on Googlebot, not Google-Extended. Blocking the AI crawler won't affect your map pack visibility or local organic rankings.

Beyond Robots.txt: Additional Protection Methods

Robots.txt relies on voluntary compliance. According to Tollbit data, 13.26% of AI bot requests ignored robots.txt directives in Q2 2025, up from 3.3% in Q4 2024.

For stronger enforcement:

Server-Level Blocking: Configure Apache .htaccess or Nginx to return 403 Forbidden responses to specific user agents.

Cloudflare AI Audit: Cloudflare's tool helps publishers monitor crawler activity and enforce blocking rules automatically.

IP Allowlisting: Google publishes IP ranges for its crawlers in JSON files. Firewall rules can verify requests against these ranges while blocking spoofed user agents.

Rate Limiting: Set limits on requests per minute to prevent aggressive crawling from overwhelming your server resources.

Frequently Asked Questions

Does blocking Google-Extended hurt my SEO rankings?

No. Google's technical documentation explicitly states that blocking Google-Extended has no impact on search rankings, indexation, or visibility. Only blocking Googlebot affects your search presence. These are separate crawlers with completely different functions.

Will blocking Google-Extended prevent my site from appearing in AI Overviews?

Blocking Google-Extended stops your content from training Gemini and Vertex AI models. However, AI Overviews in Google Search use Googlebot data. To avoid AI Overviews entirely, you would need to block Googlebot, which removes you from search results. Currently there's no way to appear in search but not in AI Overviews.

How do I know if my robots.txt is working?

Check that your file is accessible at yourdomain.com/robots.txt and use Google Search Console's robots.txt tester. Note that Google-Extended won't appear in standard server logs like Googlebot does because it functions as a control token rather than a traditional crawler.

Should small business websites bother blocking AI crawlers?

It depends on your content. Most local service businesses have limited proprietary content worth protecting. Your competitive advantage comes from service quality, reviews, and local visibility, not from preventing AI training. However, if you publish substantial original content like guides or research, blocking may make sense.

What other AI crawlers should I consider blocking?

Major training crawlers include GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), and Bytespider (ByteDance). Search-focused crawlers include PerplexityBot and OAI-SearchBot. Blocking training crawlers protects your content while search crawlers may provide visibility in AI search results.

Take Control of Your Content​

Take Control of Your Content

Understanding Google-Extended gives you an informed choice about how your website participates in AI development. The crawler's separation from search indexing means you can make this decision based on business strategy rather than SEO concerns.

Whether you block AI crawlers entirely, allow selective access, or embrace full participation, the choice should align with your content goals and revenue model. For local businesses focused on search visibility, this decision rarely impacts day-to-day operations. For publishers with substantial original content, it's increasingly becoming a strategic priority.

Need help evaluating your website's technical SEO configuration or understanding how crawlers interact with your content? Contact PushLeads for a consultation.

Who is Jeremy Ashburn?

Jeremy Ashburn has a unique blend of graphic design, web design, sales and marketing, business, and SEO experience. He’s the President and owner of Pushleads.com, a SEO Agency with the vision of “creating more traffic with less effort.” Jeremy's clients have generated Millions of dollars by doing all forms of Digital Marketing.

After college graduation, he worked for a “fast and furious” advertising agency, Jeremy worked 8 years an Executive Recruiter, and became self-taught in web design, working with Google to do SEO, doing Google Ads, Facebook Ads, Retargeting, and Pay Per Click.

In the past Fifteen years, Jeremy's created hundreds of websites, created blogs that make thousands, become a pro at ranking websites in Google, increased ROI for all of his clients, and helped his client grow dramatically.

View Success Stories

PushLeads has helped many other businesses grow with its SEO services; you’re next!

Don’t forget to review the testimonials, too. You’ll understand why PushLeads is the choice for SEO services for small businesses and mid-sized companies alike. An Asheville SEO specialist is only as good as his track record. See for yourself some of the amazing results PushLeads has accomplished through its combination of SEO services and outstanding customer service.

Hiring Pushleads is one of the best things we have done for our business. They have helped to more than triple the business we get from our website. I would highly recommend their services to anyone who is frustrated with SEO and wants to finally start making money from their website!

Diane Holmes · 1-800 Water Damage

BEST COMPANY IN TOWN!!! Let's start by letting everyone know that this company does what they say and don't blow smoke. Internet marketing is a tricky maze to navigate and as a business owner I don't have enough time to try to figure everything out. Pushleads takes all that pressure off and allows you to focus on your business. We gave Pushleads a 3 month trail period and told Jeremy we expected results. Well within 3 months we have more business then we can handle and its still growing. Pushleads is going to be a life time partner with us and you would be a fool not to sign up with them.

Caleb Wilson · All About Plumbing

I've used the team at PushLeads for over 2 years now. They consistently go above and beyond to make sure my website is ranking at the top in Google for a variety of keywords. Not only that, they’ve worked seamlessly with my website copywriters to make sure that my website is not just informative and functional but is also a net for capturing leads. I will continue to use their services and highly recommend them to other artists like myself who want to grow their online business.

Matt Tommey · Sculptural Basket Artist

Read Google Reviews

See Facebook Reviews

Browse Alignable Reviews

Video Testimonials

All About Plumbing

5 Star

Grove Manor Flooring

Video Case Study & Client Dashboard

HVAC Video Case Study

View the Client Dashboard

Watched the video above? If so, book a call below to meet with me...Email me at [email protected] if you can't find a time.


Find Those Who Need You...Reach A Larger Audience

We can help you reach a larger audience that can benefit from your services.

SEO

Make it easy for people to find you in Google. Show up in Google Maps and Searches. Results driven SEO services.

Google Ads

Facebook usage is up right now. Drive customer acquisition and revenue through target Facebook ads.

Facebook Ads

Target potential customers in a geographical location who need your services now. Outsmart your competition.

Premium Website Design

Update your business website  with a modern professional look. We do Web Design with the focus of your customer in mind. 

Budget Website Design

Whether you’re in a need of a website to get your business online, we'll bring your vision to life. Ask us about budget packages.

Marketing Workshops

Are you a Do-It-Yourselfer? Get hands on marketing training where we'll do the marketing work together to make your business more visible.

Google's latest documentation updates for the Google-Extended crawler bring practical changes that affect website owners and content managers. These modifications focus on AI training permissions and search visibility, with specific updates reflecting the transition from Bard to Gemini Apps. The changes give site owners direct control over how their content supports AI development.

A Little More About Google-Extended

Since its launch on September 28, 2023, Google-Extended has been offering web publishers a new way to manage their site's visibility. It's all about control. You can tell Google-Extended to take a hike or welcome it in, depending on whether you want your site's content to train AI models. This choice is made possible through the Robots Exclusion Protocol, where you can set the rules of engagement for this particular crawler.

The technical implementation of Google-Extended builds upon established web crawling standards. Website administrators use specific commands in their robots.txt files to set permissions. These commands act as digital barriers or welcome signs, determining whether AI training systems can access and process the site's content.

Though Google calls it a "standalone product token," that term might sound like jargon. Simply put, it's a way for you to specify how Google's AI-focused crawlers interact with your site.

The initial announcement made things clear:

"We're rolling out Google-Extended to give you more control. It's your call if you want your site to contribute to making Bard and Vertex AI's generative APIs smarter. This decision could influence how advanced these AI models get over time."

Website owners who want to restrict access can implement a simple code block in their robots.txt file:

makefile

User-agent: Google-Extended
Disallow: /

Keeping Up with Changes

Google's good at keeping a log of updates and changes, especially ones that matter to web publishers and marketing consultants. They've tweaked the Google-Extended documentation, especially after rebranding Bard to Gemini Apps. Now, Google-Extended's crawling efforts are aimed at Gemini Apps and Vertex AI, but here's the kicker: it won't mess with your Google Search standing.

The separation between AI training and search indexing represents a significant technical distinction. Search crawlers continue their regular operation, maintaining the existing SEO structures and ranking factors. Meanwhile, Google-Extended operates as an independent system, collecting data specifically for AI model training.

So, What's the Big Update?

The main takeaway from the recent changes is that Google-Extended's crawling is now focused on Gemini Apps, and it leaves Google Search rankings untouched.

In their own words:

"We've updated our terms to reflect Bard's name change to Gemini Apps. Based on your feedback, we've made it clear: Google-Extended's crawling is all about Gemini Apps and doesn't affect Google Search."

They've ditched the Bard name in favor of Gemini and added a reassuring note:

"Google-Extended won't impact how your site ranks or appears in Google Search."

Technical Implementation Details

The robots.txt configuration accepts several variations to give site owners precise control. Administrators can block specific directories while allowing others, creating a balanced approach to content sharing. For example:

User-agent: Google-Extended
Allow: /public/
Disallow: /private/

This granular control helps organizations maintain a strategic balance between contributing to AI advancement and protecting sensitive information.

Practical Applications and Benefits

Website owners now have three clear options for managing their content's role in AI development:

1. Full access: Allow Google-Extended to crawl all content, supporting AI model training across their entire site.
2. Partial access: Use directory-specific rules to share selected content while protecting other areas.
3. Complete restriction: Block Google-Extended entirely, opting out of AI training contributions.

These choices let organizations align their content strategy with their business objectives and data-sharing preferences.

Future Implications

The separation between search rankings and AI training creates new opportunities for content strategy. Organizations can now participate in AI development without concerns about search visibility impact. This change opens doors for selective content sharing based on business goals rather than SEO considerations. As companies align their content efforts with their strategic objectives, they can leverage insights from AI training to enhance their messaging and audience engagement. This shift allows for innovation in developing top SEO strategies for 2024, focusing on quality and relevance rather than solely on search algorithms. Emphasizing value-driven content will not only boost brand loyalty but also improve overall performance in search rankings over time.

For those managing a website and looking to fine-tune their exposure to AI training without affecting search visibility, these updates provide clarity and control. PushLeads highlights these changes to ensure you're in the know, especially if you're navigating the complexities of online visibility and search engine optimization as a marketing consultant.

Read Next: Essential Principles of Keyword Research

CLICK HERE to schedule your FREE consultation TODAY!

2026 Update: Navigating the AI Crawler Landscape

The roster of AI training crawlers has expanded significantly since Google-Extended launched. Site owners must now account for GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity AI), and CCBot (Common Crawl) alongside Google-Extended. Each respects its own robots.txt directive, so blocking one has no effect on the others. A comprehensive policy requires a separate disallow rule for every bot you wish to restrict.

It is equally important to understand what Google-Extended does not control. This token governs only Google's AI training pipelines — it is not a ranking signal and carries zero influence over your position in organic search results. Google's AI Overviews feature is powered by standard Googlebot, meaning pages blocked via Google-Extended can still be crawled, indexed, and surfaced inside AI Overviews without any additional configuration.

Google-Extended FAQ

Does blocking Google-Extended hurt my website's search rankings?

No, blocking Google-Extended does not affect your search rankings. It controls only whether your content trains Google's AI products like Bard and Gemini. Googlebot, which handles Search indexing, operates independently. Your pages can still be fully crawled, indexed, and ranked normally.

What is the difference between Google-Extended and Googlebot?

Googlebot crawls your site for Search indexing and ranking purposes. Google-Extended is a separate crawler token that specifically harvests content to train Google's AI models. Blocking one does not affect the other, giving site owners granular control over each use case.

How do I block Google-Extended in my robots.txt file?

Add these two lines to your robots.txt file: 'User-agent: Google-Extended' followed by 'Disallow: /'. This prevents the crawler from accessing any page. You can also target specific directories by replacing the slash with a particular path.

Does blocking Google-Extended remove my site from AI Overviews?

Not necessarily. AI Overviews can still reference content Googlebot has already indexed for Search. Blocking Google-Extended stops future AI training data collection but does not guarantee removal from AI-generated summaries, which draw from the existing Search index.

Is your business showing up in AI answers?

Run a free audit on any page in about 20 seconds.

Your path · Google Crawlers

2 articles. About 40 minutes.

  1. You’re herePart 1 · 25 minHow Google Crawler Works: Google-Extended and robots.txt
  2. Up nextPart 2 · 15 minRethinking Robots.txt: A New Method

Up next

Rethinking Robots.txt: A New Method

15 min read · Revamp your site's SEO with Gary Illyes' innovative robots.txt management strategies for more effective search engine indexing and site performance.

Want this done for you?

Scan your site. See the gaps. Decide with numbers.

  • Rankings, Google Business Profile and site-speed teardown
  • Report lands in your inbox in about a minute
  • No obligation · 4.9 from 67 Google reviews

Prefer a human? Book 20 minutes with Jeremy

Continue to Part 2

Get started

Not sure which service fits?

Three steps from first call to growth.

  1. 1Make an appointmentA short 30 minute Zoom.
  2. 2Meet with usWe review your website and current marketing, and make recommendations.
  3. 3Go!Pick a project. We meet every week online to show progress.
Free30 minutesZoomNo obligation

Calendar not loading? Open it in a new tab or call (828) 348-7686