Google Chrome Web Store Now Supports User Agents
▼ Summary
– Google has introduced a new user agent called Google-CWS for the Chrome Web Store, which is a user-triggered fetcher.
– This user agent requests URLs provided by developers in the metadata of their Chrome extensions and themes.
– User-triggered fetchers are initiated by users to perform fetching functions within Google products and typically ignore robots.txt rules.
– An example includes Google Site Verifier acting on a user’s request or a Google Cloud-hosted site retrieving an external RSS feed.
– Recognizing this user agent in crawl logs helps identify it as originating from the Chrome Web Store fetcher.
The Google Chrome Web Store now features a dedicated user agent known as Google-CWS, a development recently outlined in Google’s official help documentation. This specialized user agent functions as a user-triggered fetcher, designed to retrieve specific URLs that developers include within the metadata of their Chrome extensions and themes.
Google has clarified that user-triggered fetchers are actions initiated directly by users through Google products. These fetches occur when someone uses a feature that requires pulling external data. For instance, when a person requests Google Site Verifier to check a website, or when a site hosted on Google Cloud Platform enables visitors to fetch an external RSS feed. Because these actions stem from user commands, they typically bypass standard robots.txt directives. However, they still adhere to the general technical guidelines that govern Google’s web crawlers.
Recognizing this new user agent in your server logs is important for webmasters and developers. When you spot Google-CWS, it indicates that the Chrome Web Store is accessing the URLs you’ve listed in your extension or theme details. This insight helps you better understand the traffic sources interacting with your web properties and confirms that these visits are legitimate and tied to your Chrome Web Store listings.
(Source: Search Engine Land)
