[Skip to main content](https://developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate#main-content) * [Crawling infrastructure](https://developers.google.com/crawling) / * English * Deutsch * Español * Español – América Latina * Français * Indonesia * Italiano * Polski * Português – Brasil * Tiếng Việt * Türkçe * Русский * العربيّة * हिंदी * ภาษาไทย * 中文 – 简体 * 中文 – 繁體 * 日本語 * 한국어 Sign in [Home](https://developers.google.com/crawling) [Docs](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers) * [Crawling infrastructure](https://developers.google.com/crawling) * [Home](https://developers.google.com/crawling) * [Docs](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers) * [Intro](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers) * [About Google's web crawling](https://developers.google.com/crawling/docs/about-crawling) * How to... * [Verify requests from Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) * [Authenticate requests with Web Bot Auth (experimental)](https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth) * [Reduce Google's crawl rate](https://developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate) * Use robots.txt to manage crawling * [Create and submit a robots.txt file](https://developers.google.com/crawling/docs/robots-txt/create-robots-txt) * [How Google interprets the robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec) * [Update your robots.txt file](https://developers.google.com/crawling/docs/robots-txt/submit-updated-robots-txt) * [List of useful robots.txt rules](https://developers.google.com/crawling/docs/robots-txt/useful-robots-txt-rules) * Optimize crawling performance * [Optimize your crawl budget](https://developers.google.com/crawling/docs/crawl-budget) * [Myths about crawling](https://developers.google.com/crawling/docs/myths-about-crawling) * [Improve crawling of faceted navigation URLs](https://developers.google.com/crawling/docs/faceted-navigation) * Reference * [Common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) * [Special case crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers) * [User-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) * Specific crawlers and user-triggered fetchers * [APIs-Google](https://developers.google.com/crawling/docs/crawlers-fetchers/apis-user-agent) * [Feedfetcher](https://developers.google.com/crawling/docs/crawlers-fetchers/feedfetcher) * [Googlebot](https://developers.google.com/search/docs/crawling-indexing/googlebot) * [Google Read Aloud](https://developers.google.com/crawling/docs/crawlers-fetchers/read-aloud-user-agent) * Troubleshooting * [HTTP status codes](https://developers.google.com/crawling/docs/troubleshooting/http-status-codes) * [Network and DNS errors](https://developers.google.com/crawling/docs/troubleshooting/dns-network-errors) * What's new * [Changelog](https://developers.google.com/crawling/docs/changelog) * [Home](https://developers.google.com/) * [Crawling infrastructure](https://developers.google.com/crawling) * [Docs](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers) Send feedback Stay organized with collections Save and categorize content based on your preferences. Reduce the Google crawl rate ============================ Google's crawler infrastructure has sophisticated algorithms to determine the optimal crawl rate for a site. Our goal is to crawl as many pages from your site as we can on each visit without overwhelming your server. In some cases, Google's crawling of your site might be causing a critical load on your infrastructure, or cause unwanted costs during an outage. To alleviate this, you may decide to reduce the number of requests made by Google's crawlers. Understand the cause of the sharp increase in crawling ------------------------------------------------------ Sharp increase in crawling may be caused by inefficiencies in your site's structure or issues with your site otherwise. Based on the reports we've received in the past, the most common causes are: * Inefficient configuration of URLs on the site, which is typically casued by a specific functionality of the site: * Faceted navigation or other sorting and filtering functionality of the site * A calendar with a lot of URLs for specific dates * [A Dynamic Search Ad target](https://developers.google.com/search/docs/crawling-indexing/troubleshoot-crawling-errors#adsbot) We strongly recommend that you check with your hosting company and look at recent access logs of your server to understand the source of the traffic, and see if it fits in the aformentioned common causes of the sharp increase in crawling. Then, check our guides about [managing crawling of faceted navigation URLs](https://developers.google.com/search/docs/crawling-indexing/crawling-managing-faceted-navigation) and [optimizing crawling efficiency](https://developers.google.com/search/docs/crawling-indexing/troubleshoot-crawling-errors#improve_crawl_efficiency) . Urgently reduce crawler traffic (for emergencies) ------------------------------------------------- **Warning**: When considering reducing the Google's crawl rate, keep in mind that this will have broad effects. For Search, Googlebot will discover fewer new pages, and existing pages will be refreshed less frequently (for example, prices and product availability may take longer to be reflected in Search), and removed pages may stay in the index longer. For Google Ads, your campaigns may be cancelled or paused, and your ads may not serve. If you need to urgently reduce the crawl rate for short period of time (for example, a couple of hours, or 1-2 days), then return `500`, `503`, or `429` HTTP response status code instead of `200` to the crawl requests. Google's crawling infrastructure reduces your site's crawling rate when it encounters a significant number of URLs with `500`, `503`, or `429` HTTP response status codes (for example, if you [disabled your website](https://developers.google.com/search/docs/crawling-indexing/pause-online-business) ). The reduced crawl rate affects the whole hostname of your site (for example, `subdomain.example.com`), both the crawling of the URLs that return errors, as well as the URLs that return content. Once the number of these errors is reduced, the crawl rate will automatically start increasing again. **Warning**: We don't recommend that you do this for a long period of time (meaning, longer than 1-2 days) as it may have a negative effect on how your site appears in Google products. For example, in case of Search, if Googlebot observes these status codes on the same URL for multiple days, the URL may be dropped from Google's index. Exceptional requests to reduce crawl rate ----------------------------------------- If serving errors to Google's crawlers is not feasible on your infrastructure, [file a special request](https://search.google.com/search-console/googlebot-report) to report a problem with unusually high crawl rate, mentioning the optimal rate for your site in your request. You cannot request an increase in crawl rate, and it may take several days for the request to be evaluated and fulfilled. Send feedback Except as otherwise noted, the content of this page is licensed under the [Creative Commons Attribution 4.0 License](https://creativecommons.org/licenses/by/4.0/) , and code samples are licensed under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0) . For details, see the [Google Developers Site Policies](https://developers.google.com/site-policies) . Java is a registered trademark of Oracle and/or its affiliates. Last updated 2025-12-18 UTC. * ### Connect * [Blog](https://googledevelopers.blogspot.com/) * [Bluesky](https://goo.gle/3FReQXN) * [Instagram](https://www.instagram.com/googlefordevs/) * [LinkedIn](https://www.linkedin.com/showcase/googledevelopers/) * [X (Twitter)](https://twitter.com/googledevs) * [YouTube](https://www.youtube.com/user/GoogleDevelopers) * ### Programs * [Google Developer Program](https://developers.google.com/program) * [Google Developer Groups](https://developers.google.com/community) * [Google Developer Experts](https://developers.google.com/community/experts) * [Accelerators](https://developers.google.com/community/accelerators) * [Google Cloud & NVIDIA](https://developers.google.com/community/nvidia) * ### Developer consoles * [Google API Console](https://console.developers.google.com/) * [Google Cloud Platform Console](https://console.cloud.google.com/) * [Google Play Console](https://play.google.com/apps/publish) * [Firebase Console](https://console.firebase.google.com/) * [Actions on Google Console](https://console.actions.google.com/) * [Cast SDK Developer Console](https://cast.google.com/publish) * [Chrome Web Store Dashboard](https://chrome.google.com/webstore/developer/dashboard) * [Google Home Developer Console](https://console.home.google.com/) [![Google Developers](https://www.gstatic.com/devrel-devsite/prod/vb08cbdb02acf7f66ad9727ddfba9d81df8806422eb5dd63dba194c9c8c7997f7/developers/images/lockup-google-for-developers.svg)](https://developers.google.com/) * [Android](https://developer.android.com/) * [Chrome](https://developer.chrome.com/home) * [Firebase](https://firebase.google.com/) * [Google Cloud Platform](https://cloud.google.com/) * [Google AI](https://ai.google.dev/) * [All products](https://developers.google.com/products) * [Terms](https://developers.google.com/terms/site-terms) * [Privacy](https://policies.google.com/privacy) * [Manage cookies](https://developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate#) * English * Deutsch * Español * Español – América Latina * Français * Indonesia * Italiano * Polski * Português – Brasil * Tiếng Việt * Türkçe * Русский * العربيّة * हिंदी * ภาษาไทย * 中文 – 简体 * 中文 – 繁體 * 日本語 * 한국어