List of Google's special-case crawlers
The special-case crawlers are used by specific Google products where there's an agreement between the crawled site and the product about the crawl process. For example, AdsBot ignores the global robots.txt user agent (*) with the ad publisher's permission. The general technical properties of Google's crawlers also apply to the special-case crawlers.
The special-case crawlers may ignore robots.txt rules and so they operate from a different IP range than the common crawlers. The IP ranges are published in the special-crawlers.json object. The special-case crawlers' reverse DNS mask matches rate-limited-proxy-***-***-***-***.google.com.
The following list shows the special-case crawlers, their user agent strings as they appear in the HTTP requests, their user agent tokens for the User-agent: line in robots.txt, and the products that are affected by crawl preferences for the crawler. The list is not exhaustive, it only covers the requestors that are more likely to show up in log files and that we've received questions about.
Caution
: The user agent string can be spoofed.
Learn how to verify if a requestor is from Google
.
APIs-Google
| User-Agent in HTTP requests | APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html) |
|---|---|
| robots.txt | User-agent token in robots.txt APIs-Google The global user agent (*) is ignored. Example robots.txt group user-agent: APIs-Google allow: /archive/1Q84 disallow: /archive/ |
| User-agent token in robots.txt | APIs-Google The global user agent (*) is ignored. |
| Example robots.txt group | user-agent: APIs-Google allow: /archive/1Q84 disallow: /archive/ |
| Affected products | Crawling preferences addressed to the APIs-Google user agent affect the delivery of push notification messages by Google APIs. |
AdsBot Mobile Web
| User-Agent in HTTP requests | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*W.X.Y.Z* Mobile Safari/537.36 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html) |
|---|---|
| robots.txt | User-agent token in robots.txt AdsBot-Google-Mobile The global user agent (*) is ignored. Example robots.txt group user-agent: AdsBot-Google-Mobile allow: /archive/1Q84 disallow: /archive/ |
| User-agent token in robots.txt | AdsBot-Google-Mobile The global user agent (*) is ignored. |
| Example robots.txt group | user-agent: AdsBot-Google-Mobile allow: /archive/1Q84 disallow: /archive/ |
| Affected products | Crawling preferences addressed to the AdsBot-Google-Mobile user agent affect Google Ads' ability to check web page ad quality. |
AdsBot
| User-Agent in HTTP requests | AdsBot-Google (+http://www.google.com/adsbot.html) |
|---|---|
| robots.txt | User-agent token in robots.txt AdsBot-Google The global user agent (*) is ignored. Example robots.txt group user-agent: AdsBot-Google allow: /archive/1Q84 disallow: /archive/ |
| User-agent token in robots.txt | AdsBot-Google The global user agent (*) is ignored. |
| Example robots.txt group | user-agent: AdsBot-Google allow: /archive/1Q84 disallow: /archive/ |
| Affected products | Crawling preferences addressed to the AdsBot-Google user agent affect Google Ads' ability to check web page ad quality. |
AdSense
| User-Agent in HTTP requests | Desktop agent Mediapartners-Google Mobile agent *(Various mobile device types)* (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html) |
|---|---|
| Desktop agent | Mediapartners-Google |
| Mobile agent | *(Various mobile device types)* (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html) |
| robots.txt | User-agent token in robots.txt Mediapartners-Google The global user agent (*) is ignored. Example robots.txt group user-agent: Mediapartners-Google allow: /archive/1Q84 disallow: /archive/ |
| User-agent token in robots.txt | Mediapartners-Google The global user agent (*) is ignored. |
| Example robots.txt group | user-agent: Mediapartners-Google allow: /archive/1Q84 disallow: /archive/ |
| Affected products | Crawling preferences addressed to the Mediapartners-Google user agent affect Google AdSense. The AdSense crawler visits participating sites in order to provide them with relevant ads. |
Google-Safety
| User-Agent in HTTP requests | Google-Safety |
|---|---|
| robots.txt | The Google-Safety user agent ignores robots.txt rules. |
| Affected products | The Google-Safety user agent handles abuse-specific crawling, such as malware discovery for publicly posted links on Google properties. As such it's unaffected by crawling preferences. |
Retired special-case crawlers
The following special-case crawlers are no longer in use, and are only noted here for historical reference.
AdsBot Mobile Web
| User-Agent in HTTP requests | Mozilla/5.0 (iPhone; CPU iPhone OS 14_7_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Mobile/15E148 Safari/604.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html) |
|---|---|
| robots.txt | User-agent token in robots.txt AdsBot-Google-Mobile The global user agent (*) is ignored. |
| User-agent token in robots.txt | AdsBot-Google-Mobile The global user agent (*) is ignored. |
| Affected products | Crawling preferences addressed to the AdsBot-Google-Mobile user agent affected Google Ads' ability to check iPhone web page ad quality. |
Duplex on the web
| User-Agent in HTTP requests | Mozilla/5.0 (Linux; Android 11; Pixel 2; DuplexWeb-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.193 Mobile Safari/537.36 |
|---|---|
| robots.txt | User-agent token in robots.txt DuplexWeb-Google Duplex on the web may ignore the * wildcard. |
| User-agent token in robots.txt | DuplexWeb-Google Duplex on the web may ignore the * wildcard. |
| Affected products | Supported the Duplex on the web service. |
Google Favicon
| User-Agent in HTTP requests | Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon |
|---|---|
| robots.txt | User-agent token in robots.txt Googlebot-Image Googlebot |
| User-agent token in robots.txt | Googlebot-Image Googlebot |
Mobile Apps Android
| User-Agent in HTTP requests | AdsBot-Google-Mobile-Apps |
|---|---|
| robots.txt | User-agent token in robots.txt AdsBot-Google-Mobile-Apps The AdsBot-Google-Mobile-Apps user agent obeyed AdsBot-Google robots rules, but ignored the global user agent (*). |
| User-agent token in robots.txt | AdsBot-Google-Mobile-Apps The AdsBot-Google-Mobile-Apps user agent obeyed AdsBot-Google robots rules, but ignored the global user agent (*). |
| Affected products | Crawling preferences addressed to the AdsBot-Google-Mobile-Apps user agent affect Google Ads' ability to check Android app page ad quality. |
Web Light
| User-Agent in HTTP requests | Mozilla/5.0 (Linux; Android 4.2.1; en-us; Nexus 5 Build/JOP40D) AppleWebKit/535.19 (KHTML, like Gecko; googleweblight) Chrome/38.0.1025.166 Mobile Safari/535.19 |
|---|---|
| robots.txt | User-agent token in robots.txt googleweblight The googleweblight user agent was used only for explicit browse requests of a human visitor, and so it ignored robots.txt rules, which are used to block automated crawling requests. |
| User-agent token in robots.txt | googleweblight The googleweblight user agent was used only for explicit browse requests of a human visitor, and so it ignored robots.txt rules, which are used to block automated crawling requests. |
| Affected products | The Web Light user agent checked for the presence of the no-transform header whenever a user clicked your page in Search under appropriate conditions. |
Adapted from Google's crawling documentation: https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers