Crawl budget in 2026: who needs to worry and how to run a crawl waste audit
Crawl budget is “the set of URLs that Google can and wants to crawl,” and for most websites it is not the problem. Google’s own crawl budget guide says it is aimed at two kinds of site: large sites (1 million+ unique pages) whose content changes about weekly, and medium or larger sites (10,000+ unique pages) whose content changes daily. It adds a third signal: many URLs sitting in “Discovered - currently not indexed.” Google calls the thresholds rough, not exact.
If you run a 300-page B2B site, a slow-to-index page is almost always a quality, linking, or duplication problem, not a budget one. If you run a store, a marketplace, a publisher, or a programmatic site, crawl waste is a real cost. This post gives you the test for which camp you are in, and a crawl waste audit for when you are in the second.
What crawl budget is made of
Google splits it into two parts:
- Crawl capacity limit. How much of your server’s time Google will use. Per Google, it limits “the total amount of time your server spends holding connections open for Google.” It rises when your site is healthy and falls when you return slow responses, 5xx errors, or 429s.
- Crawl demand. How much Google wants to crawl you, based on site size, update frequency, page quality, and relevance. The levers you control are perceived inventory (how many URLs Google thinks you have), popularity, and staleness.
Budget is the lower of the two. A fast server with a bloated, low-value URL inventory still gets crawled inefficiently; a lean site on a slow server is capped by capacity.
Do you need to care? A quick test
You probably do if two or more of these are true:
- You have more than 10,000 indexable URLs and a meaningful share changes daily (prices, stock, listings, news).
- Search Console’s Page indexing report shows a large and growing count of “Discovered - currently not indexed” or “Crawled - currently not indexed” URLs.
- New or updated pages take weeks to appear, while your best pages are indexed fine.
- Your URL count in a crawler is many times larger than the number of pages you actually want ranked, because of filters, parameters, internal search, or calendar pages.
- Server logs show Googlebot spending most of its requests on URLs you would not choose to rank.
If none apply, spend the time on content quality, internal links, and a content decay audit instead.
The five-step crawl waste audit
1. Build the URL inventory you intend to rank. List the templates that deserve to be indexed (product, category, article, location) and the count of each. This is your target. Everything else Google crawls is a candidate for waste.
2. Compare it with what Google actually requests. Use your server logs if you can get them, and Search Console’s Crawl Stats report otherwise. Group Googlebot requests by URL pattern: parameters, filter combinations, internal search results, session or tracking IDs, paginated deep pages, redirected URLs, and error responses. The pattern with the biggest share of requests and the least value is your first fix.
3. Find the leaks. The usual sources, each with Google’s stated treatment:
| Leak | What Google says | Fix |
|---|---|---|
| Duplicate and near-duplicate URLs | Consolidate duplicate content so crawling goes to unique pages | Canonicals, parameter handling, one URL per page |
| Low-value URL spaces (facets, internal search) | Block unimportant URLs with robots.txt | Disallow the patterns, keep product and category URLs open (see faceted navigation in our ecommerce guide) |
| Removed pages | Return 404 or 410; a 404 is a strong signal not to crawl the URL again | Stop redirecting everything to the homepage; return a real 404 or 410 |
| Soft 404s | They “will continue to be crawled, and waste your budget” | Return a true status code, or add real content |
| Long redirect chains | Avoid them | Point links and sitemaps at the final URL |
| Stale sitemaps | Keep them current and include <lastmod> for updated content | List only canonical, indexable 200 URLs |
4. Fix in the right order. Remove or consolidate first, block second. Two cautions from Google’s guide are worth pinning to the wall: noindex does not save crawl budget, because Google still has to request a page to see the tag, and robots.txt is not for “temporarily reallocating” budget; use it only for URLs you never want crawled. Also note that blocked URLs stay in the queue longer and are recrawled once you lift the block, so a robots.txt rule is a commitment.
5. Make the remaining pages cheap to crawl. Improve server response time and support HTTP caching. Google specifically recommends supporting 304 Not Modified so it can reuse its cached copy of a page that has not changed. If you ship a lot of unchanged pages, this is the cheapest win on the list. Speed work overlaps with Core Web Vitals, though the audience is Googlebot rather than a visitor.
Re-measure two to four weeks later: the share of Googlebot requests going to target URLs, the count of “Discovered - currently not indexed” URLs, and the lag between publishing and indexing for new pages.
Programmatic and large sites: the special case
Programmatic SEO is where crawl waste most often becomes the binding constraint, because the template can generate tens of thousands of thin URLs overnight. Publish in batches, keep every page earning its place with unique data, and keep thin combinations out of the sitemap. If Google declines to index a batch, adding more pages makes the problem worse, not better.
Crawl budget and AI crawlers
Googlebot is not the only bot on your server. Answer-engine crawlers also request your pages, and a slow or error-prone site hurts you with them too. Whether you block or allow them is a separate decision; our study of Cloud 100 robots.txt files shows what software companies actually do, and our generative engine optimization guide covers how crawlability feeds AI citations. Check server logs for both bot families before deciding anything.
FAQ
Does crawl budget matter for a small website? Rarely. Google’s guide is written for sites with 10,000+ rapidly changing pages or 1 million+ pages. A small site that is not getting indexed usually has a quality, duplication, or internal linking problem.
Does noindex save crawl budget? No. Google still requests the page to see the noindex tag. To stop crawling, use robots.txt for URLs you never want crawled, or remove the URLs and return a 404 or 410.
Should I block pages in robots.txt to boost crawl budget? Only for low-value URL spaces you will never want crawled, such as filter combinations or internal search. Google advises against using robots.txt to temporarily shift budget between sections.
How do I see how Google crawls my site? Use the Crawl Stats report in Search Console for request counts, response codes, and host status, and server logs for URL-level detail. Google’s guide also points to the Page indexing report for soft 404s and discovered-but-not-indexed URLs.
Do sitemaps increase crawl budget?
No, but accurate ones help Google prioritise. Keep them to canonical, indexable URLs and include <lastmod> when content really changes.
Want someone to run it?
Our technical SEO audit covers crawl inventory, log and Crawl Stats analysis, and a prioritised fix list. Talk to us if you have a large or fast-changing site and indexing is lagging.
One copyable AI agent, every Wednesday.
Flux founder Alejandro Rioja's weekly newsletter for operators: an agent he shipped that week, one growth experiment with real numbers, and what's working across SEO and GEO right now.
Free · 1-click unsubscribe · confirm by email and you're in.
Alejandro Rioja
Operator who builds and sells marketing-focused brands. Founder of Pickleland, founder of Flux.LA, writing about AI SEO + GEO at alejandrorioja.com.
