Googlebot File Size Limits: Official Docs Updated
▼ Summary
– Google updated its documentation to move general crawler file size limits from the Googlebot page to the broader crawling infrastructure documentation.
– The Googlebot page was then updated to specify its own limits: 2 MB for supported file types and 64 MB for PDFs for Google Search.
– This change clarifies that the 15 MB default limit applies to all Google crawlers, while the Googlebot page details product-specific limits.
– The update is part of an ongoing reorganization to separate general crawling information from product-specific guidance like Google Search.
– This separation allows Google to more easily document new crawlers and reflects that its crawling infrastructure serves multiple products beyond Search.
Understanding the specific file size limits for Google’s crawlers is crucial for webmasters and SEO professionals aiming to ensure their site’s content is fully accessible for indexing. Recent updates to Google’s official documentation have clarified where to find this information, moving general crawler defaults to a centralized location while providing more precise details for Googlebot itself. This reorganization helps streamline technical guidance for those managing large websites or complex pages.
The update involves a two-part clarification. Information about default file size limits has been moved from the Googlebot page to Google’s broader crawler documentation. Google stated the original placement was not the most logical, as these limits apply to all of Google’s crawlers and fetchers, not solely Googlebot. With the general defaults now housed in the crawler documentation, the Googlebot page was updated to describe Googlebot’s own specific limits with greater precision.
The crawling infrastructure documentation now lists a 15 MB default limit for Google’s crawlers and fetchers. In contrast, the updated Googlebot page specifies a 2 MB limit for supported file types like HTML and a 64 MB limit for PDFs when crawling specifically for Google Search. The crawler overview describes the universal default, while the Googlebot page outlines the Google Search-specific limits. It’s important to remember that each resource referenced in a page’s HTML, such as CSS and JavaScript files, is fetched separately and subject to these constraints.
This documentation shift is part of a broader pattern from Google. Late last year, the company migrated its core crawling documentation to a standalone site, separating it from Search Central. The rationale was that Google’s crawling infrastructure supports numerous products beyond Search, including Shopping, News, and AdSense. Subsequent updates included guidance on faceted navigation and crawl budget optimization.
The latest change continues this reorganization effort. The 15 MB file size limit was first officially documented in 2022 when Google added it to the Googlebot help page. A Google representative confirmed then that the limit was not new but had been in effect for years; the company was simply formalizing it in the public record. For those managing crawl budgets or troubleshooting indexing issues on content-heavy pages, it’s now essential to consult the correct documentation. The limits are described differently depending on the source: the crawling infrastructure overview cites the 15 MB default, while the Googlebot page lists the 2 MB and 64 MB figures. The official changelog does not explicitly explain the relationship between these numbers.
Default limits now reside in the crawler overview documentation, while Googlebot-specific limits are detailed on the Googlebot page. This separation sets the stage for further updates to the crawling infrastructure site in the coming months. By distinguishing crawler-wide defaults from product-specific guidelines, Google can more efficiently document new crawlers and fetchers as they are developed and deployed.
(Source: Search Engine Journal)
