What crawl budget is, how Google sets it, and who should care

Google can't crawl every publicly reachable URL on the web, so it limits how much time and how many resources it spends on any one site. Google calls that allocation the site's crawl budget.

For this purpose a site is a hostname. Google's own example is a www host and a code subdomain on the same domain, which count as two sites with two separate crawl budgets.

Two elements set the budget. The first is the crawl capacity limit, also called host load. It caps how long your server spends holding connections open for Google, counting both the number of parallel connections and how long they last. Every site starts with the same moderate default.

If there is demand to crawl more and the site stays healthy, Google raises the limit over time. If responses slow down or the server returns 5xx errors or HTTP 429, the limit falls and Google crawls less.

The second is crawl demand, and each crawler has its own. AdsBot wants more when a site runs dynamic ad targeting, and Google Shopping wants more for products in merchant feeds. For Googlebot, demand depends on the site's size, how often it updates, page quality and relevance. The capacity limit is shared across all of these crawlers. Low demand also means less crawling, even when the capacity limit hasn't been reached.

Google describes its guide as advanced and aimed mainly at three kinds of site. The first is large sites, over a million unique pages, whose content changes about once a week. The second is medium or large sites, over 10,000 unique pages, whose content changes daily. The third is any site where much of its URL total sits under Discovered - currently not indexed in Search Console. Google says these numbers are rough guides, not exact limits.

If your site isn't one of these, or new pages appear to be crawled the day they go live, Google says you only need to keep your sitemap up to date and check the Page Indexing report regularly.

Google names perceived inventory as the most controllable factor. Without guidance, Google tries to crawl all or most of the URLs it finds on a site. Duplicates and pages you don't want crawled, such as removed or unimportant ones, waste that time. The guide's first recommendations follow from this. Manage the URL inventory so Google knows which pages to crawl. Consolidate duplicate content so crawling focuses on unique content, not unique URLs. Block URLs you don't need crawled with robots.txt.

Popularity and staleness also matter. URLs that are visited more across the internet tend to be crawled more often, and Google tries to recrawl documents often enough to catch changes. Site-wide events, such as a move to new URLs, can raise demand while Google reprocesses the content.