Understanding Crawl Budget and Server Capacity
A website's crawl budget is the balance between how often search engine spiders want to crawl its pages and how many requests the website's server can safely handle without degrading performance. If a site has more crawlable pages than its server can safely serve within a target update cycle, search engines may fail to discover new or updated content.
Evaluating these limits requires analyzing two distinct forces: crawl demand (how frequently search engines need to request pages to keep their indexes fresh) and safe server capacity (the volume of automated traffic the hosting infrastructure can support). Technical SEO personnel and site maintainers use these estimates to identify potential bottlenecks, plan infrastructure upgrades, and ensure that search engine spiders can crawl the entire page inventory within a target timeframe.
Estimating Daily Crawl Demand
Crawl demand is driven by the size of a website and how frequently its content changes. To estimate the daily crawl demand, the tool uses the following formula:
Daily Demand = ( Crawlable pages / Average update interval ) × Demand multiplier
In this calculation, the "Average update interval" can be specified in "days", "weeks", or "months". The "Demand multiplier" (which defaults to 1.2) acts as a buffer to account for non-uniform crawling patterns, duplicate requests, and auxiliary assets that crawlers may fetch alongside the primary HTML pages.
For example, if a site has 10,000 crawlable pages that update on average every 10 days, and we apply a demand multiplier of 1.2, the estimated "Crawl demand / day" is:
Daily Demand = ( 10,000 / 10 ) × 1.2 = 1,200 requests/day
Assessing Safe Server Capacity and Bottlenecks
A server's ability to handle crawler traffic depends on its overall performance limits and the portion of those resources allocated to automated bots. The tool evaluates safe capacity by calculating two distinct limits and selecting the lower of the two:
- The Request-Rate Limit: Based on the maximum "Server capacity (requests/second)" and the "Capacity available to crawlers (%)".
- The Concurrency Limit: Based on the "Allowed crawler concurrency" and the "Average response time (ms)".
The raw capacity is then reduced by the "Safe utilization (%)" (which defaults to 70%) to ensure that crawler activity does not overwhelm the server or slow down real user sessions.
Identifying the Capacity Bottleneck
The tool identifies what limits your server's crawling throughput by outputting one of two values for the "Capacity bottleneck":
- Available request rate: This bottleneck occurs when the overall server throughput allocated to crawlers limits the crawl rate.
- Concurrency and response time: This bottleneck occurs when slow page response times or strict limits on simultaneous connections restrict how many pages can be fetched, even if the server has spare request-rate capacity.
Input Validation and Calculation Rules
To generate an estimate, the tool requires all input fields to be completed. The calculations are governed by the following rules and constraints:
- Required Fields: If any input is missing, the tool displays the error message: "Complete every input to calculate the estimate."
- Numeric Limits: All inputs must be ordinary finite numbers within the supported range, up to a maximum allowed input value of 1e12. If an input violates this, the tool displays: "Enter ordinary finite numbers within the supported range."
- Positive Values: Every input must be greater than zero. If a zero or negative value is entered, the tool displays: "Every input must be greater than zero."
- Percentage Limits: Percentage assumptions, specifically "Capacity available to crawlers (%)" and "Safe utilization (%)", must be no more than 100%. If a value exceeds 100%, the tool displays: "Percentage assumptions must be no more than 100%."
- Coverage Status: The "Demand covered" output is visually limited to a range of 0–100%. If the calculated coverage is less than 100%, the status displayed is "The modeled demand is above the safe capacity estimate". If the coverage is 100% or more, the status displayed is "The safe capacity estimate covers the modeled demand".
Local Processing and Privacy
All calculations are performed locally. Your site and server assumptions are calculated in your browser, and nothing is uploaded to BroBroGo.
Frequently Asked Questions
Does this predict how often Googlebot will crawl my site?
No. Search engines set crawl demand and capacity from signals that are not fully public. This estimate is for planning and comparison; calibrate it with server logs and crawl reports.
How is the estimate calculated?
Daily demand is page count divided by the target update interval, then multiplied by the demand factor. Safe capacity is the lower of the request-rate and concurrency limits, reduced by the safety utilization.
What is the difference between crawl budget estimation and actual search engine crawl behavior?
An estimator uses mathematical assumptions about server limits and update frequencies to establish a safe operational baseline. Actual search engine crawling is dynamic and depends on factors like content value, server errors, search demand, and proprietary crawler policies.
Why does response time affect the concurrency limit?
If a server takes longer to respond to a single request, a crawler must hold connections open for a longer duration. If there is a strict limit on "Allowed crawler concurrency", slow response times will drastically reduce the total number of pages a crawler can fetch per day.