Google's Bot Selection
Google has updated its “Verifying Googlebot and other Google crawlers” support document, adding a new section that clearly explains its three types of crawlers: Googlebot, special-case crawlers, and user-triggered crawlers.
Some, including myself, were curious about the recently noticed GoogleOther crawler, which led to this update. In response to our interest, Gary Illyes of Google remarked, “Please don’t overthink it; it’s quite straightforward.” Yet, my further inquiries motivated Gary to expand on the topic through the support document.
Google’s crawlers are categorized into three distinct groups, providing a clearer understanding of how Google indexes the web. Let’s dive in and get to know the three different types of Google crawlers.
1. Googlebot
This is the principal crawler for Google’s search offerings and always complies with robots.txt rules. Its reverse DNS mask is “crawl—-.googlebot.com or geo-crawl—-.geo.googlebot.com,” and its IP range list can be found in the googlebot.json file.
As the main crawler, Googlebot’s primary function is to gather information from websites to build an index for Google’s search engine. It’s strictly compliant with robots.txt rules, which means it respects the instructions given by a website on what content can be crawled and what should be left alone.
2. Special-Case Crawlers
These crawlers carry out specific tasks like the AdsBot and may or may not adhere to robots.txt rules. The reverse DNS mask is “rate-limited-proxy—-.google.com,” The list of IP ranges is accessible in the special-crawlers.json file.
These are more specialized bots tasked with specific functions. Unlike Googlebot, these crawlers may or may not adhere to the rules set out in a website’s robots.txt file. This means they might crawl areas of a website that the Googlebot would not. An example is AdsBot, which is designed to crawl websites for Google Ads purposes.
3. User-Triggered Fetchers
These are tools and product functions initiated by the end user, like the Google Site Verifier. As the fetch is user-requested, these fetchers overlook robots.txt rules. The reverse DNS mask is “—.gae.googleusercontent.com”, and the list of IP ranges can be viewed in the user-triggered-fetchers.json file.
These are unique in that they’re initiated by user action rather than running autonomously like the other two types. Since these fetchers are user-requested, they ignore robots.txt rules, potentially fetching data that other crawlers wouldn’t access. A prime example is the Google Site Verifier, used to confirm website ownership.
This further understanding of Google’s different crawlers is an important development for anyone invested in optimizing their website’s visibility and performance in Google’s search results. As always, staying updated with Google’s changes is key to ensuring your website is fully optimized for its crawlers.
Moreover, in conjunction with this update, Google has reorganized the sequence of Google bots on the crawlers’ webpage.
Read Next: 6 Essential Questions You Need to Know on How to Achieve Keyword Research for Effective SEO.
For SEO audits, Click Here to book your appointment.
What’s Your SEO Score?
Enter the URL of any landing page or blog article and see how optimized it is for one keyword or phrase.
