[Skip to main content](https://developers.google.com/search/docs/crawling-indexing/googlebot#main-content) [![Google Search Central](https://developers.google.com/static/search/images/google-search-central-logo.svg)](https://developers.google.com/search) * [Google Search Central](https://developers.google.com/search) [Documentation](https://developers.google.com/search/docs) * SEO fundamentals * [Introduction](https://developers.google.com/search/docs) * [Search Essentials](https://developers.google.com/search/docs/essentials) * [SEO Starter Guide](https://developers.google.com/search/docs/fundamentals/seo-starter-guide) * [Optimizing for generative AI search](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) * [How Google Search Works](https://developers.google.com/search/docs/fundamentals/how-search-works) * [Do you need an SEO?](https://developers.google.com/search/docs/fundamentals/do-i-need-seo) * Crawling and indexing * [Sitemaps](https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview) * [robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro) * [Meta tags](https://developers.google.com/search/docs/crawling-indexing/special-tags) * [Crawler management](https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl) * [Removals](https://developers.google.com/search/docs/crawling-indexing/control-what-you-share) * [Canonicalization](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls) * [Redirects](https://developers.google.com/search/docs/crawling-indexing/301-redirects) * [JavaScript SEO](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) * Ranking and search appearance * [Visual Elements gallery](https://developers.google.com/search/docs/appearance/visual-elements-gallery) * [Title links](https://developers.google.com/search/docs/appearance/title-link) * [Snippets](https://developers.google.com/search/docs/appearance/snippet) * [Images](https://developers.google.com/search/docs/appearance/google-images) * [Videos](https://developers.google.com/search/docs/appearance/video) * [Structured data](https://developers.google.com/search/docs/appearance/structured-data/search-gallery) * [Favicons](https://developers.google.com/search/docs/appearance/favicon-in-search) * Site-specific guides * [Ecommerce](https://developers.google.com/search/docs/specialty/ecommerce) * [International and multilingual sites](https://developers.google.com/search/docs/specialty/international) * Data analysis * [Get started with Search Console](https://developers.google.com/search/docs/monitor-debug/search-console-start) * [Debug traffic drops](https://developers.google.com/search/docs/monitor-debug/debugging-search-traffic-drops) * [Using Search Console and Google Analytics data for SEO](https://developers.google.com/search/docs/monitor-debug/google-analytics-search-console) * [Get started with Google Trends](https://developers.google.com/search/docs/monitor-debug/trends-start) [Support](https://developers.google.com/search/help) [Blog](https://developers.google.com/search/blog) [What's new](https://developers.google.com/search/news) * [All updates](https://developers.google.com/search/news) * [Documentation updates](https://developers.google.com/search/updates) * [Ranking updates](https://status.search.google.com/products/rGHU1u87FJnkP6W2GwMi/history) * [New YouTube videos](https://www.youtube.com/channel/UCWf2ZlNsCGDS89VBF_awNvA) * [Recent podcast episodes](https://pod.link/1512522198) [Events](https://developers.google.com/search/events) [Case studies](https://developers.google.com/search/case-studies) / * English * Deutsch * Español * Español – América Latina * Français * Indonesia * Italiano * Polski * Português – Brasil * Tiếng Việt * Türkçe * Русский * العربيّة * हिंदी * ภาษาไทย * 中文 – 简体 * 中文 – 繁體 * 日本語 * 한국어 Sign in * [Documentation](https://developers.google.com/search/docs) [Search Console](https://goo.gle/searchconsole) [![Google Search Central](https://developers.google.com/static/search/images/google-search-central-logo.svg)](https://developers.google.com/search) * [Google Search Central](https://developers.google.com/search) * [Documentation](https://developers.google.com/search/docs) * More * [Support](https://developers.google.com/search/help) * [Blog](https://developers.google.com/search/blog) * [What's new](https://developers.google.com/search/news) * More * [Events](https://developers.google.com/search/events) * [Case studies](https://developers.google.com/search/case-studies) * [Introduction](https://developers.google.com/search/docs) * Search Essentials * [Overview](https://developers.google.com/search/docs/essentials) * [Technical requirements](https://developers.google.com/search/docs/essentials/technical) * [Spam policies](https://developers.google.com/search/docs/essentials/spam-policies) * SEO fundamentals * [SEO Starter Guide](https://developers.google.com/search/docs/fundamentals/seo-starter-guide) * [How Google Search works](https://developers.google.com/search/docs/fundamentals/how-search-works) * [Creating helpful, reliable, people-first content](https://developers.google.com/search/docs/fundamentals/creating-helpful-content) * Generative AI fundamentals * [Optimizing for generative AI](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) * [Guidance on using generative AI](https://developers.google.com/search/docs/fundamentals/using-gen-ai-content) * [Maintaining your site's SEO](https://developers.google.com/search/docs/fundamentals/get-started) * [Developer's guide to Search](https://developers.google.com/search/docs/fundamentals/get-started-developers) * [Do you need an SEO?](https://developers.google.com/search/docs/fundamentals/do-i-need-seo) * Crawling and indexing * [Overview](https://developers.google.com/search/docs/crawling-indexing) * [File types Google can index](https://developers.google.com/search/docs/crawling-indexing/indexable-file-types) * [URL structure](https://developers.google.com/search/docs/crawling-indexing/url-structure) * [Links](https://developers.google.com/search/docs/crawling-indexing/links-crawlable) * Sitemaps * [Learn about sitemaps](https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview) * [Build and submit a sitemap](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap) * [Manage sitemaps with sitemap index file](https://developers.google.com/search/docs/crawling-indexing/sitemaps/large-sitemaps) * Sitemap extensions * [Image sitemaps](https://developers.google.com/search/docs/crawling-indexing/sitemaps/image-sitemaps) * [News sitemaps](https://developers.google.com/search/docs/crawling-indexing/sitemaps/news-sitemap) * [Video sitemaps and alternatives](https://developers.google.com/search/docs/crawling-indexing/sitemaps/video-sitemaps) * [Combining sitemap extensions](https://developers.google.com/search/docs/crawling-indexing/sitemaps/combine-sitemap-extensions) * Crawler management * [Ask Google to recrawl your URLs](https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl) * [Troubleshoot crawl errors](https://developers.google.com/search/docs/crawling-indexing/troubleshoot-crawling-errors) * Google crawlers * [List of Google crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers) * [Googlebot](https://developers.google.com/search/docs/crawling-indexing/googlebot) * [Reduce the Google crawl rate](https://developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate) * [Verifying Googlebot and other Google crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/verifying-googlebot) * robots.txt * [Introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro) * [How Google interprets the robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec) * Canonicalization * [What is URL canonicalization](https://developers.google.com/search/docs/crawling-indexing/canonicalization) * [How to specify a canonical URL with rel="canonical" and other methods](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls) * [Fix canonicalization issues](https://developers.google.com/search/docs/crawling-indexing/canonicalization-troubleshooting) * [Mobile site and mobile-first indexing](https://developers.google.com/search/docs/crawling-indexing/mobile/mobile-sites-mobile-first-indexing) * AMP * [AMP on Google Search guidelines](https://developers.google.com/search/docs/crawling-indexing/amp) * [Understand how AMP works in search results](https://developers.google.com/search/docs/crawling-indexing/amp/about-amp) * [Enhance AMP content](https://developers.google.com/search/docs/crawling-indexing/amp/enhance-amp) * [Validate AMP content](https://developers.google.com/search/docs/crawling-indexing/amp/validate-amp) * [Remove AMP content](https://developers.google.com/search/docs/crawling-indexing/amp/remove-amp) * JavaScript * [Understand the JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) * [Fix search-related JavaScript problems](https://developers.google.com/search/docs/crawling-indexing/javascript/fix-search-javascript) * [Fix lazy-loaded content](https://developers.google.com/search/docs/crawling-indexing/javascript/lazy-loading) * [Dynamic rendering as a workaround](https://developers.google.com/search/docs/crawling-indexing/javascript/dynamic-rendering) * Page and content metadata * [Page metadata](https://developers.google.com/search/docs/crawling-indexing/valid-page-metadata) * Meta tags * [Meta tags and HTML attributes that Google supports](https://developers.google.com/search/docs/crawling-indexing/special-tags) * [Robots meta tag, data-nosnippet, and X-Robots-Tag](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag) * [noindex](https://developers.google.com/search/docs/crawling-indexing/block-indexing) * [rel attributes](https://developers.google.com/search/docs/crawling-indexing/qualify-outbound-links) * Removals * [Control what you share with Google](https://developers.google.com/search/docs/crawling-indexing/control-what-you-share) * [Page removals](https://developers.google.com/search/docs/crawling-indexing/remove-information) * [Image removals](https://developers.google.com/search/docs/crawling-indexing/prevent-images-on-your-page) * [Redacted information](https://developers.google.com/search/docs/crawling-indexing/keep-redacted-information-out) * Site moves and changes * [Redirects and Google Search](https://developers.google.com/search/docs/crawling-indexing/301-redirects) * Site moves * [Changing your hosting](https://developers.google.com/search/docs/crawling-indexing/site-move-no-url-changes) * [Move a site with URL changes](https://developers.google.com/search/docs/crawling-indexing/site-move-with-url-changes) * [A/B testing](https://developers.google.com/search/docs/crawling-indexing/website-testing) * [Temporarily pause or disable a website](https://developers.google.com/search/docs/crawling-indexing/pause-online-business) * Ranking and search appearance * [Overview](https://developers.google.com/search/docs/appearance) * [AI features](https://developers.google.com/search/docs/appearance/ai-features) * [Byline dates](https://developers.google.com/search/docs/appearance/publication-dates) * [Favicons](https://developers.google.com/search/docs/appearance/favicon-in-search) * [Featured snippets](https://developers.google.com/search/docs/appearance/featured-snippets) * [Flexible Sampling](https://developers.google.com/search/docs/appearance/flexible-sampling) * [Google Discover](https://developers.google.com/search/docs/appearance/google-discover) * [Images](https://developers.google.com/search/docs/appearance/google-images) * Local features * [Business details](https://developers.google.com/search/docs/appearance/establish-business-details) * [Top Places List](https://developers.google.com/search/docs/appearance/top-places-list) * [Opting out of local search results](https://support.google.com/webmasters/answer/3035947) * Page experience * [Understanding page experience](https://developers.google.com/search/docs/appearance/page-experience) * [Core Web Vitals](https://developers.google.com/search/docs/appearance/core-web-vitals) * [Interstitials and dialogs](https://developers.google.com/search/docs/appearance/avoid-intrusive-interstitials) * [Get started with signed exchanges on Google Search](https://developers.google.com/search/docs/appearance/signed-exchange) * [Preferred sources](https://developers.google.com/search/docs/appearance/preferred-sources) * Ranking systems * [A guide to Google Search ranking systems](https://developers.google.com/search/docs/appearance/ranking-systems-guide) * [Reviews system](https://developers.google.com/search/docs/appearance/reviews-system) * Ranking updates * [List of ranking updates](https://status.search.google.com/products/rGHU1u87FJnkP6W2GwMi/history) * [Core updates](https://developers.google.com/search/docs/appearance/core-updates) * [Spam updates](https://developers.google.com/search/docs/appearance/spam-updates) * [Site names](https://developers.google.com/search/docs/appearance/site-names) * [Sitelinks](https://developers.google.com/search/docs/appearance/sitelinks) * [Snippets](https://developers.google.com/search/docs/appearance/snippet) * Structured data * [Understand how structured data works](https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data) * [Structured data general guidelines](https://developers.google.com/search/docs/appearance/structured-data/sd-policies) * [Enriched search results](https://developers.google.com/search/docs/appearance/enriched-search-results) * [Generate structured data with JavaScript](https://developers.google.com/search/docs/appearance/structured-data/generate-structured-data-with-javascript) * Feature guides * [All structured data features](https://developers.google.com/search/docs/appearance/structured-data/search-gallery) * [Article](https://developers.google.com/search/docs/appearance/structured-data/article) * [Book actions](https://developers.google.com/search/docs/appearance/structured-data/book) * [Breadcrumb](https://developers.google.com/search/docs/appearance/structured-data/breadcrumb) * [Carousel](https://developers.google.com/search/docs/appearance/structured-data/carousel) * [Course list](https://developers.google.com/search/docs/appearance/structured-data/course) * [Dataset](https://developers.google.com/search/docs/appearance/structured-data/dataset) * [Discussion forum](https://developers.google.com/search/docs/appearance/structured-data/discussion-forum) * [Education Q&A](https://developers.google.com/search/docs/appearance/structured-data/education-qa) * [Employer aggregate rating](https://developers.google.com/search/docs/appearance/structured-data/employer-rating) * [Fact check](https://developers.google.com/search/docs/appearance/structured-data/factcheck) * [Event](https://developers.google.com/search/docs/appearance/structured-data/event) * [FAQ](https://developers.google.com/search/docs/appearance/structured-data/faqpage) * [Image metadata](https://developers.google.com/search/docs/appearance/structured-data/image-license-metadata) * [Job posting](https://developers.google.com/search/docs/appearance/structured-data/job-posting) * [Local business](https://developers.google.com/search/docs/appearance/structured-data/local-business) * [Math solver](https://developers.google.com/search/docs/appearance/structured-data/math-solvers) * [Movie carousel](https://developers.google.com/search/docs/appearance/structured-data/movie) * [Organization](https://developers.google.com/search/docs/appearance/structured-data/organization) * Shopping * [Overview](https://developers.google.com/search/docs/appearance/structured-data/product) * [Product snippet](https://developers.google.com/search/docs/appearance/structured-data/product-snippet) * [Merchant listing](https://developers.google.com/search/docs/appearance/structured-data/merchant-listing) * [Variants](https://developers.google.com/search/docs/appearance/structured-data/product-variants) * [Loyalty program](https://developers.google.com/search/docs/appearance/structured-data/loyalty-program) * [Merchant return policy](https://developers.google.com/search/docs/appearance/structured-data/return-policy) * [Merchant shipping policy](https://developers.google.com/search/docs/appearance/structured-data/shipping-policy) * [Profile page](https://developers.google.com/search/docs/appearance/structured-data/profile-page) * [Q&A](https://developers.google.com/search/docs/appearance/structured-data/qapage) * [Recipe](https://developers.google.com/search/docs/appearance/structured-data/recipe) * [Review snippet](https://developers.google.com/search/docs/appearance/structured-data/review-snippet) * [Software app](https://developers.google.com/search/docs/appearance/structured-data/software-app) * [Speakable](https://developers.google.com/search/docs/appearance/structured-data/speakable) * [Subscription and paywalled content](https://developers.google.com/search/docs/appearance/structured-data/paywalled-content) * [Vacation rental](https://developers.google.com/search/docs/appearance/structured-data/vacation-rental) * [Video](https://developers.google.com/search/docs/appearance/structured-data/video) * [Title links](https://developers.google.com/search/docs/appearance/title-link) * Translated features * [Translated results](https://developers.google.com/search/docs/appearance/translated-results) * [Ad networks and translation-related Google Search features](https://developers.google.com/search/docs/appearance/ad-network-and-translation) * [Videos](https://developers.google.com/search/docs/appearance/video) * [Visual Elements gallery](https://developers.google.com/search/docs/appearance/visual-elements-gallery) * Web Stories * [Enable Web Stories on Google](https://developers.google.com/search/docs/appearance/enable-web-stories) * [Best practices for creating Web Stories](https://developers.google.com/search/docs/appearance/web-stories-creation-best-practices) * [Web Story content policy](https://developers.google.com/search/docs/appearance/web-stories-content-policy) * Early Adopters Program * [Package tracking](https://developers.google.com/search/docs/appearance/package-tracking) * [Structured data carousels (beta)](https://developers.google.com/search/docs/appearance/structured-data/carousels-beta) * Monitoring and debugging * [Debug drops in Search traffic](https://developers.google.com/search/docs/monitor-debug/debugging-search-traffic-drops) * Monitor with Search Console * [Get started with Search Console](https://developers.google.com/search/docs/monitor-debug/search-console-start) * [Improve SEO with a bubble chart](https://developers.google.com/search/docs/monitor-debug/bubble-chart-analysis) * [Using Search Console and Google Analytics data for SEO](https://developers.google.com/search/docs/monitor-debug/google-analytics-search-console) * Debug with search operators * [Overview](https://developers.google.com/search/docs/monitor-debug/search-operators) * [site: search operator](https://developers.google.com/search/docs/monitor-debug/search-operators/all-search-site) * [Google Images search operators](https://developers.google.com/search/docs/monitor-debug/search-operators/image-search) * Preventing and monitoring abuse * [Overview](https://developers.google.com/search/docs/monitor-debug/security) * [Prevent user-generated spam](https://developers.google.com/search/docs/monitor-debug/prevent-abuse) * [Malware and unwanted software](https://developers.google.com/search/docs/monitor-debug/security/malware) * [Prevent a malware infection](https://developers.google.com/search/docs/monitor-debug/security/prevent-malware) * [Social engineering (phishing and deceptive sites)](https://developers.google.com/search/docs/monitor-debug/security/social-engineering) * [Google Safe Browsing Repeat Offenders Policy](https://developers.google.com/search/docs/monitor-debug/security/safe-browsing-repeat-offenders) * [Get started with Google Trends](https://developers.google.com/search/docs/monitor-debug/trends-start) * Site-specific guides * Ecommerce * [Overview](https://developers.google.com/search/docs/specialty/ecommerce) * [Where content can appear](https://developers.google.com/search/docs/specialty/ecommerce/where-ecommerce-data-can-appear-on-google) * [Share product data](https://developers.google.com/search/docs/specialty/ecommerce/share-your-product-data-with-google) * [Include structured data](https://developers.google.com/search/docs/specialty/ecommerce/include-structured-data-relevant-to-ecommerce) * [Launch a new website](https://developers.google.com/search/docs/specialty/ecommerce/how-to-launch-an-ecommerce-website) * [Write high quality reviews](https://developers.google.com/search/docs/specialty/ecommerce/write-high-quality-reviews) * [Design a URL structure](https://developers.google.com/search/docs/specialty/ecommerce/designing-a-url-structure-for-ecommerce-sites) * [Ecommerce site structure](https://developers.google.com/search/docs/specialty/ecommerce/help-google-understand-your-ecommerce-site-structure) * [Pagination, incremental page loading, and Search](https://developers.google.com/search/docs/specialty/ecommerce/pagination-and-incremental-page-loading) * International and multilingual * [Overview](https://developers.google.com/search/docs/specialty/international) * [Managing multi-regional and multilingual sites](https://developers.google.com/search/docs/specialty/international/managing-multi-regional-sites) * [Tell Google about localized versions of your page](https://developers.google.com/search/docs/specialty/international/localized-versions) * [How Google crawls locale-adaptive pages](https://developers.google.com/search/docs/specialty/international/locale-adaptive-pages) * Explicit content * [Guidelines for sites with explicit content](https://developers.google.com/search/docs/specialty/explicit/guidelines) * [What to do if your site is incorrectly flagged as explicit](https://developers.google.com/search/docs/specialty/explicit/troubleshooting) * SEO fundamentals * [Introduction](https://developers.google.com/search/docs) * [Search Essentials](https://developers.google.com/search/docs/essentials) * [SEO Starter Guide](https://developers.google.com/search/docs/fundamentals/seo-starter-guide) * [Optimizing for generative AI search](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) * [How Google Search Works](https://developers.google.com/search/docs/fundamentals/how-search-works) * [Do you need an SEO?](https://developers.google.com/search/docs/fundamentals/do-i-need-seo) * Crawling and indexing * [Sitemaps](https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview) * [robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro) * [Meta tags](https://developers.google.com/search/docs/crawling-indexing/special-tags) * [Crawler management](https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl) * [Removals](https://developers.google.com/search/docs/crawling-indexing/control-what-you-share) * [Canonicalization](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls) * [Redirects](https://developers.google.com/search/docs/crawling-indexing/301-redirects) * [JavaScript SEO](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) * Ranking and search appearance * [Visual Elements gallery](https://developers.google.com/search/docs/appearance/visual-elements-gallery) * [Title links](https://developers.google.com/search/docs/appearance/title-link) * [Snippets](https://developers.google.com/search/docs/appearance/snippet) * [Images](https://developers.google.com/search/docs/appearance/google-images) * [Videos](https://developers.google.com/search/docs/appearance/video) * [Structured data](https://developers.google.com/search/docs/appearance/structured-data/search-gallery) * [Favicons](https://developers.google.com/search/docs/appearance/favicon-in-search) * Site-specific guides * [Ecommerce](https://developers.google.com/search/docs/specialty/ecommerce) * [International and multilingual sites](https://developers.google.com/search/docs/specialty/international) * Data analysis * [Get started with Search Console](https://developers.google.com/search/docs/monitor-debug/search-console-start) * [Debug traffic drops](https://developers.google.com/search/docs/monitor-debug/debugging-search-traffic-drops) * [Using Search Console and Google Analytics data for SEO](https://developers.google.com/search/docs/monitor-debug/google-analytics-search-console) * [Get started with Google Trends](https://developers.google.com/search/docs/monitor-debug/trends-start) * [All updates](https://developers.google.com/search/news) * [Documentation updates](https://developers.google.com/search/updates) * [Ranking updates](https://status.search.google.com/products/rGHU1u87FJnkP6W2GwMi/history) * [New YouTube videos](https://www.youtube.com/channel/UCWf2ZlNsCGDS89VBF_awNvA) * [Recent podcast episodes](https://pod.link/1512522198) * [Home](https://developers.google.com/) * [Search Central](https://developers.google.com/search) * [Documentation](https://developers.google.com/search/docs) Send feedback Googlebot ========= Googlebot is the generic name for two types of [web crawlers](https://developers.google.com/search/docs/fundamentals/how-search-works) used by Google Search: * [**Googlebot Smartphone**](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers#googlebot-smartphone) : a mobile crawler that simulates a user on a mobile device. * [**Googlebot Desktop**](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers#googlebot-desktop) : a desktop crawler that simulates a user on desktop. You can identify the subtype of Googlebot by looking at the [HTTP `user-agent` request header](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers) in the request. However, both crawler types obey the same product token (user agent token) in robots.txt, and so you cannot selectively target either Googlebot Smartphone or Googlebot Desktop using robots.txt. For most sites Google Search primarily [indexes the mobile version](https://developers.google.com/search/docs/crawling-indexing/mobile/mobile-sites-mobile-first-indexing) of the content. As such the majority of Googlebot crawl requests will be made using the mobile crawler, and a minority using the desktop crawler. How Googlebot accesses your site -------------------------------- For most sites, Googlebot shouldn't access your site more than once every few seconds on average. However, due to delays it's possible that the rate will appear to be slightly higher over short periods. If your site is having trouble keeping up with Google's crawling requests, you can [reduce the crawl rate.](https://developers.google.com/search/docs/crawling-indexing/reduce-crawl-rate) When crawling for Google Search, Googlebot crawls the first 2MB of a [supported file type](https://developers.google.com/search/docs/crawling-indexing/indexable-file-types) , and the first 64MB of a PDF file. From a rendering perspective, each resource referenced in the HTML (such as CSS and JavaScript) is fetched separately, and each resource fetch is bound by the same file size limit that applies to other files (except PDF files). Once the cutoff limit is reached, Googlebot stops the fetch and only sends the already downloaded part of the file for indexing consideration. The file size limit is applied on the uncompressed data. Other Google crawlers, for example Googlebot Video and Googlebot Image, may have [different limits](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers#file-size-limits) . When crawling from IP addresses in the US, the timezone of Googlebot is [Pacific Time](https://g.co/kgs/WSf8oR) . Other [technical properties of Googlebot](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers#crawl-technical-props) are described in the overview of Google's crawlers. Blocking Googlebot from visiting your site ------------------------------------------ Googlebot discovers new URLs to crawl primarily from links embedded in previously crawled pages. It's almost impossible to keep a site secret by not publishing links to it. For example, as soon as someone clicks a link from your "secret" site to another site, your "secret" site URL may appear in the referrer tag and can be stored and published by the other site in its referrer log. If you want to prevent Googlebot from crawling content on your site, you have a [number of options](https://developers.google.com/search/docs/crawling-indexing/control-what-you-share) . Remember there's a difference between _crawling_ and _indexing_; blocking Googlebot from crawling a page doesn't prevent the URL of the page from appearing in search results: * **Prevent Googlebot from crawling a page?** Use a [robots.txt file](https://developers.google.com/search/docs/crawling-indexing/robots/intro) . * **Don't want Google to index a page?** Use [`noindex`](https://developers.google.com/search/docs/crawling-indexing/block-indexing) . * **Prevent a page from being accessible at all by both crawlers or users?** Use [another method, such as password protection](https://developers.google.com/search/docs/crawling-indexing/control-what-you-share) . Blocking Googlebot affects Google Search (including Discover and all Google Search features), as well as other products such as Google Images, Google Video, and Google News. Verifying Googlebot ------------------- Before you decide to block Googlebot, be aware that the HTTP `user-agent` request header used by Googlebot is often spoofed by other crawlers. It's important to verify that a problematic request actually comes from Google. The best way to verify that a request actually comes from Googlebot is to [use a reverse DNS lookup](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot#manual) on the source IP of the request, or to match the source IP against the [Googlebot IP ranges](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot#use-automatic-solutions) . Send feedback Except as otherwise noted, the content of this page is licensed under the [Creative Commons Attribution 4.0 License](https://creativecommons.org/licenses/by/4.0/) , and code samples are licensed under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0) . For details, see the [Google Developers Site Policies](https://developers.google.com/site-policies) . Java is a registered trademark of Oracle and/or its affiliates. Last updated 2026-02-03 UTC. * [![LinkedIn](https://developers.google.com/static/search/images/li.png)LinkedIn](https://www.linkedin.com/showcase/googlesearchcentral/) Join us on LinkedIn * [![YouTube](https://developers.google.com/static/homepage-assets/images/yt.svg)YouTube](https://www.youtube.com/channel/UCWf2ZlNsCGDS89VBF_awNvA) Watch our videos * [![Blog](https://fonts.gstatic.com/s/i/short-term/release/googlesymbols/rss_feed/default/24px.svg)Blog](https://feeds.feedburner.com/blogspot/amDG) Subscribe to our RSS feed * [![Podcast](https://developers.google.com/static/search/images/search-off-the-record-podcast-logo.png)Podcast](https://pod.link/1512522198) Listen to Search Off the Record * [![X (Twitter)](https://developers.google.com/static/homepage-assets/images/x.svg)X (Twitter)](https://twitter.com/googlesearchc) Join us on X (Twitter) * ### Get support * [Go to the help forum](https://support.google.com/webmasters/community) * [Submit a question for office hours](https://developers.google.com/search/help/office-hours) * [Report spam, phishing, or malware](https://developers.google.com/search/help/report-quality-issues) * [More support resources](https://developers.google.com/search/help) * ### Resources * [Do you need an SEO?](https://developers.google.com/search/docs/fundamentals/get-on-google) * [SEO Starter Guide](https://developers.google.com/search/docs/fundamentals/seo-starter-guide) * [Status of Search systems](https://status.search.google.com/) * [Search Console documentation](https://support.google.com/webmasters) * [Case Studies](https://developers.google.com/search/case-studies) * ### Tools * [Search Console](https://search.google.com/search-console) * [Rich Results Test](https://search.google.com/test/rich-results) * [PageSpeed Insights](https://pagespeed.web.dev/) * [AMP Test](https://search.google.com/test/amp) [![Google Developers](https://www.gstatic.com/devrel-devsite/prod/vb08cbdb02acf7f66ad9727ddfba9d81df8806422eb5dd63dba194c9c8c7997f7/developers/images/lockup-google-for-developers.svg)](https://developers.google.com/) * [Android](https://developer.android.com/) * [Chrome](https://developer.chrome.com/home) * [Firebase](https://firebase.google.com/) * [Google Cloud Platform](https://cloud.google.com/) * [Google AI](https://ai.google.dev/) * [All products](https://developers.google.com/products) * [Terms](https://developers.google.com/terms/site-terms) * [Privacy](https://policies.google.com/privacy) * [Manage cookies](https://developers.google.com/search/docs/crawling-indexing/googlebot#) * English * Deutsch * Español * Español – América Latina * Français * Indonesia * Italiano * Polski * Português – Brasil * Tiếng Việt * Türkçe * Русский * العربيّة * हिंदी * ภาษาไทย * 中文 – 简体 * 中文 – 繁體 * 日本語 * 한국어