Googlebot Leads 2025 Web Crawling Amid AI Bot Surge

A new analysis of web traffic confirms that Googlebot remains the most active web crawler by a significant margin, even as activity from artificial intelligence bots skyrockets. According to Cloudflare’s latest annual review, Google’s primary crawler generated more traffic than any other automated agent throughout 2025, solidifying its central role in both search indexing and AI model training. This dominance persists despite a massive fifteen-fold year-over-year increase in crawling by bots designed to mimic human actions for AI data collection.

The data reveals that Googlebot was responsible for over a quarter of all traffic from verified bots. More strikingly, it accounted for 4.5% of all HTML requests on Cloudflare’s network. This figure alone exceeds the combined total of 4.2% from all other AI-focused crawlers. The volume of requests from Googlebot far surpassed that of prominent AI crawlers from companies like OpenAI, Anthropic, and Meta.

This surge in AI-related web crawling has not gone unnoticed by website administrators. AI crawlers were the user agents most frequently blocked entirely via robots.txt files, indicating widespread concern over how these bots access and use content. Among these AI platforms, Anthropic’s crawler exhibited an exceptionally high crawl-to-referral ratio. This metric compares how much content a bot accesses to how much traffic it sends back to websites. Early in the year, Anthropic’s ratio approached 500,000 to one, meaning it crawled vast amounts of data without directing significant human visitors. This ratio later stabilized between 25,000:1 and 100,000:1 after May.

Other AI platforms showed different patterns. OpenAI’s ratio saw a spike to around 3,700:1 in March. In contrast, Perplexity maintained the lowest ratio among major AI crawlers. It began the year below 100:1, experienced a brief increase above 700:1 during a crawl spike in late March, and then remained mostly under 400:1, often falling below 200:1 from September onward.

The behavior of traditional search engine crawlers presented a very different picture. Microsoft’s Bingbot operated with a consistent weekly cycle, maintaining a crawl-to-referral ratio between 50:1 and 70:1. Google’s ratio showed more fluctuation, rising from just over 3:1 to about 30:1 by April, then falling back to around 3:1 by mid-summer before gradually increasing again. DuckDuckGo’s crawler stayed below a 1:1 ratio for most of the year, indicating it sent more referral traffic than it crawled, before jumping to approximately 1.5:1 in October and staying at that elevated level.

This crawling activity underpins a search engine market where Google maintains a near-monopoly on referral traffic. The tech giant delivered close to 90% of all search engine referrals observed by Cloudflare throughout the year. The competitive landscape saw minimal change, with Bing a distant second at 3.1%, followed by Yandex at 2.0%, Baidu at 1.4%, and DuckDuckGo at 1.2%. Market shares remained largely stable, with slight movements like Yandex dipping from 2.5% to 1.5% between May and July, and Baidu rising from 0.9% to 1.6% from April to June.

These insights come from Cloudflare’s comprehensive “2025 Radar Year in Review” report, which also examined trends in post-quantum cryptography adoption and recorded distributed denial-of-service (DDoS) attacks. The data collectively illustrates a web ecosystem where Google’s infrastructure is more pervasive than ever, even as a new generation of AI bots rapidly expands its footprint across the internet.

(Source: Search Engine Land)