Google’s Gary Illyes has introduced an innovative approach to managing robots.txt files that could change the way we handle site crawling configurations, particularly with CDNs involved. The modification comes at a time when web architectures are becoming increasingly complex, with multiple servers and content delivery systems working in tandem. This new method addresses long-standing challenges in managing crawl directives across distributed networks while maintaining search engine accessibility.
A Shift in Traditional Thinking
Traditionally, it’s been assumed that a robots.txt file must be stationed at the root domain—for instance, at example.com/robots.txt. However, in a surprising update via LinkedIn, Illyes explained that this is not strictly necessary. This revelation challenges years of established SEO practices and opens new possibilities for website optimization. The traditional placement requirement has often created bottlenecks in content delivery and complicated server configurations, especially for sites using multiple domains or complex hosting arrangements.
In this video
- 0:00 Google-Extended Crawler What It Is
- 1:00 Why 94 Percent of Sites Have No Rules
- 1:46 Google-Extended vs Googlebot Key Differences
- 2:42 robots.txt Six-Step Setup Guide
- 3:29 AI Crawler Referral Ratios Why They Matter
- 4:40 Publisher Blocking Trends and Data
- 5:35 Your Content Is a Real Business Asset
- 6:11 Other AI Crawlers and Their Tokens
- 7:25 Verifying Your robots.txt in GSC
Show transcript
Your robots.txt file is making decisions about AI data access right now, whether you wrote those rules intentionally or not. We're walking through exactly what Google-Extended is, how it differs from the crawler controlling your search rankings, and how to make a deliberate choice about who trains AI models on your content. This is PushLeads, and this is your complete guide to Google-Extended.
Quick pause for context. I'm Jeremy with PushLeads, and helping local business owners and marketers win in search is what I do all day. My first sites hit Google's first page back in 2007, and I never stopped.
Only 5.6% of the top 10,000 domains have explicit rules for Google-Extended in their robots.txt files. Roughly 94 out of every 100 websites have made no deliberate choice about this crawler. They're open by default, and default is not a strategy — it's just the absence of one.
Google announced Google-Extended in September 2023 to give publishers a clean, documented way to separate AI training access from search crawling. Before that, there was no easy mechanism to say yes to search indexing and no to AI data collection simultaneously.
Google-Extended reads your published pages and sends that content to Google to train Gemini and future AI products. It's completely separate from Googlebot, uses a different user-agent token, and follows its own robots.txt rules. It's an officially documented, fully supported feature.
Googlebot and Google-Extended are two different crawlers with two completely different jobs. Googlebot crawls and indexes your site for Google Search. It affects your rankings and drives referral traffic. Blocking it removes you from search results entirely. Google-Extended does none of that. Its only job is collecting content for AI model training. It has zero effect on your search rankings, your Maps listing, or any other Google search product. Blocking it is completely safe. The two follow separate directives in your robots.txt, and blocking one has absolutely no effect on the other.
Every website already has a robots.txt file talking to every crawler that visits, including Google-Extended. If you've never added an explicit rule for it, the file isn't staying neutral. Silence means open access — no rule equals full permission. The product token is simply Google-Extended. Two lines in your robots.txt puts you in complete control of that pipeline without touching anything related to search.
Step one: find your robots.txt file at your domain slash robots.txt and open it in any text editor. Step two: add the user-agent block — type "User-agent: Google-Extended" on its own line. Step three: write the disallow directive. "Disallow: /" blocks the entire site, or use specific paths like /blog/ to protect only certain sections. Step four: save and upload the file, replacing the existing version at your domain root. Step five: open Google Search Console, find the robots.txt report under Settings, and confirm the file reads exactly as you wrote it. Step six: use the URL Inspection tool to verify Googlebot can still reach your pages.
The crawl-to-referral ratio for AI training crawlers is staggering. Google Search runs at roughly 14-to-one — 14 pages crawled for every visit returned through rankings. That's a reasonable exchange. Anthropic's crawler ran at 73,000-to-one in June 2025, according to Cloudflare data. 73,000 pages consumed for every single referral visit returned to publishers. For any business depending on website traffic to generate leads, that math matters enormously.
The publisher response is measurable and growing fast. Cloudflare's analysis of the top 10,000 domains shows GPT Bot is the most frequently blocked AI crawler, with 31.2% of those domains blocking it. Google-Extended ranks third at 5.6% of analyzed robots.txt files. ClaudeBot is now blocked by approximately 5.8 million websites, up from 3.2 million as recently as July 2025.
Publishers have figured out that targeted AI crawler control is real, effective, and carries no search ranking risk. Harry Clarkson-Bennett, SEO director at The Telegraph, put it plainly: publishers are blocking AI bots using robots.txt because there is almost no value exchange. Large language models are not designed to send referral traffic, and publishers still need traffic to survive.
That logic applies equally to a local contractor, a boutique, a photographer, or any small business whose website content drives revenue. 79% of top US and UK news websites now block at least one AI training crawler. The overall blocking rate increased 336% over the prior year, according to Tolbit's Q2 2025 report. ClaudeBot alone crossed 5.8 million blocked sites in a matter of months. This is not a fringe concern. It's a mainstream business decision publishers across every size and sector are actively making.
A handwritten guide about local renovation costs, a detailed service page explaining your process, a portfolio of completed projects — that content took real expertise and real time to produce. AI training systems find exactly that kind of specific, locally grounded content valuable. They consume it, train on it, and send no traffic back to the business that created it. If your website generates leads, your content is a real business asset. Making a deliberate decision about who can train AI models on that asset is part of protecting your business, not just a technical detail.
Google-Extended is one crawler in a much larger ecosystem. GPT Bot is OpenAI's training crawler and the most frequently blocked across the web. CCBot is Common Crawl's bot, used by multiple AI companies, and the second most blocked. ClaudeBot is Anthropic's crawler for Claude AI. Bytespider is ByteDance's crawler, associated with TikTok's parent company. Each uses its own user-agent token, meaning each needs its own separate disallow block. A catch-all disallowing everything would block Googlebot too, removing your site from search entirely. Be specific with every token.
One common question: does blocking Google-Extended remove you from AI Overviews? Probably not directly. AI Overviews appear to pull from content Googlebot has already indexed. Google-Extended feeds model training pipelines. Those are separate processes. Google has not publicly documented a direct link between blocking Google-Extended and exclusion from AI Overviews. The reliable path to AI-generated search features is well-structured content with schema markup, clear headings, and direct answers near the top of your pages, with Googlebot able to access everything.
Writing the robots.txt rules is step one. Confirming they work as expected is step two. Open Google Search Console, find the robots.txt report under Settings, and confirm the file reads exactly as you wrote it. Check for conflicting rules — the specific block wins over a catch-all. Use the URL Inspection tool to confirm Googlebot can still reach your pages. To verify a visitor claiming to be Google-Extended, run a reverse DNS lookup. The hostname should end in googlebot.com or google.com. Then run a forward DNS lookup to confirm it returns the original IP.
If you're not confident your file is correct, a technical SEO audit shows exactly what every crawler sees.
Every AI crawler and its exact token: Google-Extended for Google's AI training — safe to block, no ranking impact. GPT Bot for OpenAI — most frequently blocked across the web. ClaudeBot for Anthropic — 5.8 million sites block it. CCBot for Common Crawl — second most blocked overall. Bytespider for ByteDance — requires its own separate block.
The golden rule: never use a catch-all "User-agent: * / Disallow: /" — that one line blocks everything, including Googlebot, and removes your site from search entirely. List each AI crawler with its own explicit block and keep your Googlebot rules completely separate.
Your robots.txt is making AI access decisions for your business right now, whether you intended it to or not. Google-Extended, GPT Bot, ClaudeBot, and others visit sites across the web every single day. Most small business sites have no explicit rules for any of them, which means they're open by default. A clear robots.txt strategy lets you keep full Google Search visibility while deciding exactly which AI systems can train on the content you worked hard to create. That is a real business decision.
See exactly where your visibility stands with a free teardown. Find the link in the description or call 828-348-7686.
Flexible File Placement
Illyes shared that it’s perfectly acceptable to have two robots.txt files—one on your main website and another on your Content Delivery Network (CDN). He suggests centralizing the robots.txt on the CDN and redirecting requests from the main site to this file. This strategy helps unify crawl directives in one place while still covering the entire web presence. The redirection process occurs seamlessly, with search engines following the redirect to access the centralized rules. This approach proves particularly effective for sites serving content across multiple regions through CDN edge locations, ensuring consistent crawler behavior regardless of access point.
Want more customers from Google & AI search?
Get a free SEO audit of your site — see exactly what to fix first.
Why This Matters
This method is especially helpful for websites that operate across multiple domains or utilize CDNs extensively. By centralizing the robots.txt file, site administrators can ensure consistent crawl rules are applied across all parts of their digital landscape, reducing the chances of miscommunication with search engines. For enterprise-level websites managing numerous subdomains and microsites, this centralization simplifies maintenance and reduces the risk of conflicting directives. The approach also enables faster updates to crawl rules, as changes need only be made in one location to affect the entire network.
Celebrating 30 Years of Robots.txt
As we mark the 30th anniversary of the Robots Exclusion Protocol (REP), Illyes’ insights reflect the protocol’s adaptability and its ongoing evolution in web standards. He even hints that future modifications could include renaming the traditional “robots.txt” file. The protocol has grown from a simple text file to a sophisticated system supporting pattern matching, wildcards, and specific user-agent directives. Its evolution mirrors the web’s development, adapting to handle modern challenges like content syndication, dynamic rendering, and multi-platform content delivery.
Benefits of Illyes' Approach
Adopting Illyes’ method offers several advantages:
- Centralized Management: Keep all your crawl directives in one place with a single, centralized robots.txt file, making updates easier and more consistent. This centralization reduces maintenance overhead and ensures immediate implementation of crawl rule changes across all properties.
- Consistency and Clarity: Minimize the risk of conflicting directives between your main site and CDN. This unified approach prevents search engines from encountering contradictory instructions that could impact site crawling and indexing.
- Adaptability: This flexible setup is ideal for complex site architectures or for businesses leveraging multiple subdomains and CDNs. The system adapts easily to network changes, new domain additions, and evolving content delivery strategies.
Implementation Considerations
When implementing this new approach, several technical aspects require attention. Server configurations must properly handle the redirection of robots.txt requests, ensuring minimal latency and maintaining proper HTTP status codes. CDN caching settings should be optimized to balance quick response times with the need for occasional updates. Additionally, monitoring systems should track both the main domain and CDN robots.txt access patterns to verify proper functionality.
Future Implications
This shift in robots.txt management signals broader changes in how search engines interact with websites. As web architecture continues to evolve, traditional assumptions about file placement and server structure require reassessment. The flexibility introduced by Illyes’ method paves the way for more efficient crawl management in increasingly complex web environments.
See also: Exploring Website Crawling: Understanding, Significance, and Optimization Strategies
This strategy simplifies management and can enhance your local SEO services by ensuring more effective and uniform site crawling rules across all your digital assets.
CLICK HERE to schedule your FREE consultation TODAY!
Source:
You Don’t Need Robots.txt On Root Domain, Says Google
Keep Reading
Want more customers from Google & AI search?
Get a free SEO audit of your site — see exactly what to fix first.
