Glossary
Crawl Budget
The number of URLs a search or AI crawler will fetch from your site in a given window — finite, so wasted crawls delay indexing of the pages that matter.
By Teeming Chew, Founder Last updated
Crawl budget is the practical limit on how many pages a crawler fetches per visit, set by your site's authority, speed, and health. On a small site it's rarely binding; on a large or slow one, crawlers can spend it on low-value URLs and miss the pages you want cited.
What wastes crawl budget?
Infinite faceted-filter URLs, duplicate pages without a canonical, redirect chains, slow responses, and thin or soft-404 pages. Each wasted fetch is one fewer for a page that deserves indexing.
Does crawl budget affect AI citation?
Indirectly but really. AI engines cite what's indexed; if a crawler never reaches your best answer page, it can't be retrieved or cited. Efficient internal linking and a clean sitemap steer budget toward high-value URLs.
How do you optimize crawl budget?
Fix duplicates and redirects, keep responses fast, prune or noindex low-value pages, and expose a current sitemap. IndexNow further reduces reliance on discovery crawling.
Part of the Cite Hustle GEO glossary — definitions for generative engine optimization and AI search. See how it fits the bigger picture in the GEO methodology.