Google’s Secret Crawler Army: Hundreds Undocumented
▼ Summary
– The term “Googlebot” is a historical misnomer, as it now refers to hundreds of different crawlers across Google’s products, not a single system.
– Googlebot is actually just one client that interacts with a larger, internal crawling infrastructure, which operates like a software service with API endpoints.
– Google’s crawling infrastructure has an internal name and functions to fetch web content responsibly based on site restrictions and specified parameters.
– Many internal Google crawlers are not publicly documented, with only major or high-volume ones being listed due to practical limitations on documentation space.
– A key distinction exists between crawlers, which process a continuous stream of URLs in batches, and fetchers, which retrieve individual URLs on a user-controlled basis.
The inner workings of Google’s web crawling system are far more complex than the single “Googlebot” name suggests. In a recent discussion, Google’s Gary Illyes and Martin Splitt peeled back the curtain, revealing that Google operates hundreds of distinct crawlers across its various products and services, most of which are not publicly documented. The term “Googlebot” is a historical artifact from a simpler time when a single crawler existed; today, it refers broadly to a vast and intricate infrastructure.
Gary Illyes clarified that “Googlebot” is not the crawling infrastructure itself. Instead, it’s more accurate to think of it as one client among many that interacts with a much larger, internal crawling service. This core infrastructure, which has an internal codename at Google, functions like a software-as-a-service (SaaS) platform. It provides API endpoints that different internal products and teams can call upon to fetch content from the web. Developers specify parameters like user agent and robots.txt directives, and the system handles the request, all with the overarching goal of fetching data “without breaking the internet.”
A key takeaway is the sheer scale of undocumented crawling activity. Illyes explained that because Google is such a large organization, numerous internal teams utilize this infrastructure for their own needs, leading to dozens or even hundreds of specialized crawlers. Documenting every single one is not feasible, so Google focuses its public documentation on major, high-volume crawlers. Smaller, low-volume crawlers often remain undocumented unless their activity grows to a significant level that warrants review and inclusion.
The conversation also distinguished between two types of automated visitors: crawlers and fetchers. Crawlers operate in batch mode, processing a continuous stream of URLs over time. Fetchers, on the other hand, work on an individual URL basis, typically triggered by a user action where someone is waiting for a response. Both types fall under the broad Googlebot umbrella, and many internal versions of each exist without public documentation.
To manage this ecosystem, Illyes mentioned an internal monitoring tool. It alerts him when any crawler or fetcher crosses a specific threshold of daily activity. This allows his team to investigate, understand the purpose of the traffic, and ensure it’s not causing unintended issues. If a crawler’s activity becomes substantial enough to be noticeable on the web, a decision is made about whether to add it to the public documentation, helping site owners understand the source of the traffic.
(Source: Search Engine Journal)
