Topic: crawl-to-refer ratio

Googlebot Leads 2025 Web Crawling Amid AI Bot Surge

Googlebot remained the most dominant web crawler in 2025, generating over a quarter of all verified bot traffic and exceeding the combined total of all major AI-focused crawlers. AI crawlers were the most frequently blocked by websites, with Anthropic's bot showing an exceptionally high crawl-to-...