Home / Crawlers & Fetchers
List of Google's common crawlers
Google's common crawlers are used to find information for building Google's search indexes, perform other product specific crawls, and for analysis. They always obey robots.txt rules when crawling automatically. The general technical properties of Google's crawlers also apply to the common crawlers.
The common crawlers generally crawl from the IP ranges published in the common-crawlers.json object, and the reverse DNS mask of their hostname matches crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com.
The following list shows the common crawlers, their user agent strings as they appear in the HTTP requests, their user agent tokens for the User-agent: line in robots.txt, and the products that are affected by crawl preferences for the crawler. Some crawlers have more than one user agent token; you need to match only one crawler token for a rule to apply. The list is not exhaustive, it only covers the requestors that are more likely to show up in log files and that we've received questions about.
Caution
: The HTTP user agent string can be spoofed.
Learn how to verify if a visitor is a Google crawler
.
User-Agent in HTTP requests | Googlebot Smartphone Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*W.X.Y.Z* Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Googlebot Desktop Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/*W.X.Y.Z* Safari/537.36 Rarely: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Googlebot/2.1 (+http://www.google.com/bot.html) |
|---|
| Googlebot Smartphone | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*W.X.Y.Z* Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
| Googlebot Desktop | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/*W.X.Y.Z* Safari/537.36 Rarely: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Googlebot/2.1 (+http://www.google.com/bot.html) |
| robots.txt | User-agent token in robots.txt Googlebot Example robots.txt group user-agent: Googlebot allow: /archive/1Q84 disallow: /archive |
| User-agent token in robots.txt | Googlebot |
| Example robots.txt group | user-agent: Googlebot allow: /archive/1Q84 disallow: /archive |
| Affected products | Crawling preferences addressed to the Googlebot user agent affect Google Search (including Discover and all Google Search features), as well as other products such as Google Images, Google Video, Google News, and Discover. |
Googlebot Image
| User-Agent in HTTP requests | Googlebot-Image/1.0 |
|---|
| robots.txt | User-agent token in robots.txt Googlebot-Image Googlebot Example robots.txt group user-agent: Googlebot-Image allow: /archive/1Q84 disallow: /archive/moons.jpg |
| User-agent token in robots.txt | Googlebot-Image Googlebot |
| Example robots.txt group | user-agent: Googlebot-Image allow: /archive/1Q84 disallow: /archive/moons.jpg |
| Affected products | Crawling preferences addressed to the Googlebot-Image user agent affect Google Images, Discover, Google Video, and all features in Google Search where images, logos, and favicons are presented. |
Googlebot Video
| User-Agent in HTTP requests | Googlebot-Video/1.0 |
|---|
| robots.txt | User-agent token in robots.txt Googlebot-Video Googlebot Example robots.txt group user-agent: Googlebot-Video allow: /archive/1Q84 disallow: /archive/ |
| User-agent token in robots.txt | Googlebot-Video Googlebot |
| Example robots.txt group | user-agent: Googlebot-Video allow: /archive/1Q84 disallow: /archive/ |
| Affected products | Crawling preferences addressed to the Googlebot-Video user agent affect video-related Google Search features and other products dependent on videos. |
Googlebot News
| User-Agent in HTTP requests | Googlebot-news doesn't have a separate HTTP request user agent string. Crawling is done with various Googlebot user agent strings. |
|---|
| robots.txt | User-agent token in robots.txt Googlebot-News Googlebot Example robots.txt group user-agent: Googlebot-News allow: /archive/1Q84 disallow: /archive/ |
| User-agent token in robots.txt | Googlebot-News Googlebot |
| Example robots.txt group | user-agent: Googlebot-News allow: /archive/1Q84 disallow: /archive/ |
| Affected products | Crawling preferences addressed to the Googlebot-News user agent affect the Google News product, including news.google.com and the Google News app. |
| User-Agent in HTTP requests | Desktop agent Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*W.X.Y.Z* Safari/537.36 Mobile agent Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*W.X.Y.Z* Mobile Safari/537.36 |
|---|
| Desktop agent | Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*W.X.Y.Z* Safari/537.36 |
| Mobile agent | Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*W.X.Y.Z* Mobile Safari/537.36 |
| robots.txt | User-agent token in robots.txt Storebot-Google Example robots.txt group user-agent: Storebot-Google allow: /archive/1Q84 disallow: /archive/konbini |
| User-agent token in robots.txt | Storebot-Google |
| Example robots.txt group | user-agent: Storebot-Google allow: /archive/1Q84 disallow: /archive/konbini |
| Affected products | Crawling preferences addressed to the Storebot-Google user agent affect all surfaces of Google Shopping (for example, the Shopping tab in Google Search and Google Shopping). |
Google-InspectionTool
| User-Agent in HTTP requests | Desktop agent Mozilla/5.0 (compatible; Google-InspectionTool/1.0;) Mobile agent Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*W.X.Y.Z* Mobile Safari/537.36 (compatible; Google-InspectionTool/1.0;) |
|---|
| Desktop agent | Mozilla/5.0 (compatible; Google-InspectionTool/1.0;) |
| Mobile agent | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*W.X.Y.Z* Mobile Safari/537.36 (compatible; Google-InspectionTool/1.0;) |
| robots.txt | User-agent token in robots.txt Google-InspectionTool Googlebot Example robots.txt group user-agent: Google-InspectionTool allow: /archive/1Q84 disallow: /archive/ |
| User-agent token in robots.txt | Google-InspectionTool Googlebot |
| Example robots.txt group | user-agent: Google-InspectionTool allow: /archive/1Q84 disallow: /archive/ |
| Affected products | Crawling preferences addressed to the Google-InspectionTool user agent affect Search testing tools such as the Rich Result Test and URL inspection in Search Console. It has no effect on Google Search or other products. |
GoogleOther
| User-Agent in HTTP requests | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*W.X.Y.Z* Mobile Safari/537.36 (compatible; GoogleOther) Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/*W.X.Y.Z* Safari/537.36 |
|---|
| robots.txt | User-agent token in robots.txt GoogleOther Example robots.txt group user-agent: GoogleOther allow: /archive/1Q84 disallow: /archive/ |
| User-agent token in robots.txt | GoogleOther |
| Example robots.txt group | user-agent: GoogleOther allow: /archive/1Q84 disallow: /archive/ |
| Affected products | Crawling preferences addressed to the GoogleOther user agent don't affect any specific product. GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development. |
GoogleOther-Image
| User-Agent in HTTP requests | GoogleOther-Image/1.0 |
|---|
| robots.txt | User-agent token in robots.txt GoogleOther-Image GoogleOther Example robots.txt group user-agent: GoogleOther-Image allow: /archive/1Q84 disallow: /archive/moon.jpg |
| User-agent token in robots.txt | GoogleOther-Image GoogleOther |
| Example robots.txt group | user-agent: GoogleOther-Image allow: /archive/1Q84 disallow: /archive/moon.jpg |
| Affected products | Crawling preferences addressed to the GoogleOther-Image user agent don't affect any specific product, similar to GoogleOther. GoogleOther-Image is the version of GoogleOther optimized for fetching publicly accessible image URLs. |
GoogleOther-Video
| User-Agent in HTTP requests | GoogleOther-Video/1.0 |
|---|
| robots.txt | User-agent token in robots.txt GoogleOther-Video GoogleOther Example robots.txt group user-agent: GoogleOther-Video allow: /archive/1Q84 disallow: /archive |
| User-agent token in robots.txt | GoogleOther-Video GoogleOther |
| Example robots.txt group | user-agent: GoogleOther-Video allow: /archive/1Q84 disallow: /archive |
| Affected products | Crawling preferences addressed to the GoogleOther-Video user agent don't affect any specific product, similar to GoogleOther. GoogleOther-Video is the version of GoogleOther optimized for fetching publicly accessible video URLs. |
Google-CloudVertexBot
| User-Agent substring in HTTP requests | Google-CloudVertexBot |
|---|
| robots.txt | User-agent token in robots.txt Google-CloudVertexBot Googlebot Example robots.txt group user-agent: Google-CloudVertexBot allow: /archive/1Q84 disallow: /archive/ |
| User-agent token in robots.txt | Google-CloudVertexBot Googlebot |
| Example robots.txt group | user-agent: Google-CloudVertexBot allow: /archive/1Q84 disallow: /archive/ |
| Affected products | Crawling preferences addressed to the Google-CloudVertexBot user agent affect crawls requested by the site owners' for building Vertex AI Agents. It has no effect on Google Search or other products. |
Google-Extended
| User-Agent in HTTP requests | Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity. |
|---|
| robots.txt | User-agent token in robots.txt Google-Extended Example robots.txt group user-agent: Google-Extended allow: /archive/1Q84 disallow: /archive/ |
| User-agent token in robots.txt | Google-Extended |
| Example robots.txt group | user-agent: Google-Extended allow: /archive/1Q84 disallow: /archive/ |
| Affected products | Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI. Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search. |
A note about Chrome/*W.X.Y.Z* in user agents
The string **Chrome/*W.X.Y.Z*** in the user agent strings in the list is a placeholder that represents the version of the Chrome browser used by that user agent: for example, 41.0.2272.96. This version number increases over time to match the latest Chromium release version used by Googlebot.
If you are searching your logs or filtering your server for a user agent with this pattern, use wildcards for the version number rather than specifying an exact version number.
Adapted from Google's crawling documentation: https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers