The shop was slow at night. Not during a sale, not at peak hour — at just past three in the morning, when almost nobody orders. CPU load looked harmless, the database did too, and yet response times climbed to a multiple of the norm. It only became visible when someone sorted the access logs by user agent instead of by time. Most requests in that window came from automated clients. And they ran almost exclusively on filter and sort URLs that no cache had ever seen. Every single one occupied a PHP worker and a database connection. The few real customers waited in the same queue.
Most requests no longer come from humans
Automated traffic is no longer a fringe phenomenon. Cloudflare reports that as of June 2026, more than 50 percent (Cloudflare Radar) of Internet traffic is no longer human — a threshold crossed for the first time. Snapshots of the Radar dashboard in June 2026 put the bot share of HTML requests at roughly 57.5 percent (Cloudflare Radar) against 42.5 percent human traffic. These readings fluctuate considerably by time window and region; they are a trend signal, not a fixed measurement for your project.
The shift happened quickly. At the start of 2025, non-AI bots still accounted for roughly half (Cloudflare Radar) of HTML requests, seven percentage points above human traffic. By early December 2025, the human share stood at 47 percent (Cloudflare Radar) and non-AI bots at 44 percent. In parallel, the category that fetches websites on behalf of a specific user question grew more than 15-fold (Cloudflare Radar) during 2025.
For the server side, absolute volume matters more than share. Across the network, 32 percent (Cloudflare) of traffic comes from automated clients, and self-identified AI bots alone generate over 10 billion requests per week (Cloudflare). Of those, around 80 percent (Cloudflare) are crawlers in the narrow sense — clients that systematically work through page after page rather than following a single user question.
The bot share is not a law of nature
A capacity topic, not an SEO topic
Why crawlers hit your most expensive URLs
A cache lives on one assumption: that many requests hit the same URL. AI crawlers break precisely that assumption. Cloudflare describes how pages once considered long tail and rarely accessed are now being requested regularly (Cloudflare). Public statistics from Common Crawl show that over 90 percent (Common Crawl) of pages are unique by content. For a crawler pursuing completeness, repetition is worthless — it looks for what it does not have yet.
The rate of unique access is correspondingly high: AI agents typically reach 70 to 100 percent (Cloudflare) unique fetches per iteration. Tests show a clear drop in cache hit rate as soon as AI crawler traffic is added (Cloudflare). Every cache miss has the same consequence: the request passes through to the origin, occupies a PHP worker there, opens a database connection and blocks both for the duration of the computation.
The URL classes affected are exactly those a shop generates technically without ever having deliberately published them:
- Facet and filter combinations such as
?colour=blue&size=42&sort=price— every combination is its own URL with its own cache entry. - Deep pagination, page 47 of a category for instance, which a human practically never reaches but a crawler walks through systematically.
- Internal search result pages, which create a new address per search term and usually go straight to the database uncached.
- Sort variants of the same list, delivering identical content in a different order and multiplying the cache.
- Session, tracking and campaign parameters, which change nothing about the content but make every cache key unique.
- Calendar and archive pages with an endless date space in which a crawler can keep going without a stop condition.
The effect is multiplicative, not additive. Five filter dimensions with four values each and three sort options produce over 3,000 URL variants per category by arithmetic alone. At 200 categories that is more than 600,000 addresses that exist technically but that hardly any human ever calls up. For the cache, those are 600,000 cold entries. For a crawler, they are 600,000 targets — and it has no reason to skip them.
Media bandwidth is affected too
Making the load visible
Before you regulate, measure. The first cut runs on the user agent — but that is freely chosen and therefore no proof. Legitimate crawlers can be verified via reverse DNS: resolve the IP to a hostname, resolve that hostname back to an IP, and check that both match. In addition, the large operators publish IP ranges you can check against. Anything that claims to be a known crawler but fails this test belongs in its own category — and gets treated differently from a verified bot.
Segment by user agent
Split access logs into agent groups: verified search engines, verified AI crawlers, unverified self-declaration, humans. Four segments are enough to start with.
Verify via reverse DNS
Forward-confirmed reverse DNS plus a check against published IP ranges. Without this step you are regulating on a claim that any client can set freely.
Evaluate TTFB separately
Response time per segment and per URL class instead of one overall average. Only that separation shows whether bot load slows the purchase path or merely itself.
Log the cache status
Record hit, miss and bypass per request. The hit rate per segment is the most direct answer to the question of who really loads the origin.
Correlate worker saturation
Plot occupied PHP workers and open database connections over time against the bot rate. Saturation explains latency jumps more reliably than CPU utilisation.
Build crawl-to-referral
Weigh pages crawled per platform against the visitors that same platform sends back. This is the metric that justifies throttling — or refutes it.
A blind spot appears if you look only at field data from the browser. Real user monitoring measures humans by definition — bots do not execute JavaScript and simply never appear there. A site can look unremarkable in field data and still be running at its saturation limit at the origin. You see bot load on the server side only. How field and lab data complement each other is covered in the article on RUM, CrUX and performance budgets.
Evaluate separately, not as an average
Crawl-to-referral: the cost-benefit ratio
The metric weighs the number of pages crawled against the visitors a platform sends back. It answers the question that comes first with any throttling decision: what does this bot cost, and what does it deliver? The spread between platforms is wider than most operators expect.
| Crawler | Pages crawled per referral (July 2025) | Reading |
|---|---|---|
| ClaudeBot (Anthropic) | 38,066 : 1 | Very high crawl load, almost no referrals |
| GPTBot (OpenAI) | 1,091 : 1 | High load, few referrals |
| PerplexityBot | 195 : 1 | Moderate load, more referrals |
| Microsoft | 41 : 1 | Comparatively balanced ratio |
All values come from measurements taken in July 2025 (Cloudflare Radar) — and they move fast. In late June 2025, Anthropic's ratio was still close to 71,000 to 1 (Cloudflare Radar); by July 2025 it stood at 38,066 to 1 (Cloudflare Radar), a decline of roughly 87 percent against January. Perplexity moved the other way over the same period, rising from 54.6 to 194.8 to 1 (Cloudflare Radar). Anyone using this metric should therefore collect it on a rolling basis rather than once.
The ratio also depends on the industry. In the news and publications category, Anthropic stood at 2,500 to 1 (Cloudflare Radar) in early August 2025, OpenAI at 152 to 1 (Cloudflare Radar) and Perplexity at 32.7 to 1 (Cloudflare Radar) — well below the overall average across all industries. For your own project, the same ratio can be built from access logs and referrer data; it is the most solid basis for deciding which crawler you serve generously and which one you limit.
A crawler that reads tens of thousands of pages and sends back a single visitor is not a marketing channel. It is a cost factor you dimension deliberately.
Throttle instead of block
The obvious reaction is a blanket ban. It is rarely the best one. The share of pure training crawls among AI crawler requests stood at 52 percent (Cloudflare Radar) in June 2026, up from 22 percent (Cloudflare Radar) in spring 2025. At the same time, over 36 percent (Cloudflare Radar) comes from mixed-use crawlers that combine search, agent use and training. That very mixture makes blanket blocking risky: block the crawler and you may also block the answer engine that would have sent visitors.
The trade-off has a real downside. In the most heavily crawled categories, human traffic has fallen by as much as 40 percent (Cloudflare Radar) in less than a year. Anyone seeking visibility in answer engines therefore needs a differentiated rule rather than a switch — and that rule should hang on the metric, not on gut feeling.
| Measure | Effect on server load | Effect on visibility |
|---|---|---|
| Blanket blocking by user agent | Load drops immediately and sharply | Useful crawlers are lost as well |
| Rate limit per verified crawler | Load is capped predictably | Content stays reachable |
| Prioritisation: humans before bots | Peaks hit the bots first | Largely unchanged |
| Cache rules for bot traffic | Cache misses go down | Unchanged, content stays retrievable |
| Remove facets from the crawl path | Crawl surface shrinks substantially | Relevant pages stay indexable |
| Send 429 instead of 503 | Relieves load without an error signal | Crawler throttles and returns |
In practice it is usually a handful of well-chosen interventions that absorb most of the load:
- Rate limits per verified bot group rather than globally, so one aggressive crawler does not consume everyone else's budget.
- A dedicated cache key for bot traffic that ignores tracking and session parameters, turning many cold URLs into a few warm ones.
- Longer cache lifetimes for bot requests wherever slightly stale content is acceptable — prices, stock levels and the basket stay excluded.
- Separate worker pools or connection limits, so bot load cannot starve the purchase path.
- 429 with
Retry-Afterinstead of 503, because a server error lowers the crawl budget over the longer term. - Prioritisation at the edge: under overload, delay bot requests first, not customer sessions.
Which cache layer is responsible for what is covered in the article on Varnish and Redis; how far load can be shifted to the edge at all is shown in the article on CDN and edge caching. Which combination fits your project is outlined on the caching strategies service page.
robots.txt as a capacity instrument
The Robots Exclusion Protocol is standardised as RFC 9309 (IETF) — considerably more precise than its reputation as a mere suggestion implies. Four details matter for capacity planning, and they are regularly overlooked in practice:
- Parsing limit: crawlers must parse at least 500 kibibytes (RFC 9309) of the file. Rules beyond that boundary may be ignored — automatically generated, very long robots.txt files are therefore risky.
- Caching: a cached robots.txt should not be used for more than 24 hours (RFC 9309). A new rule therefore does not take effect immediately but with up to a day's delay. As an emergency measure under acute load it is of little use.
- Status codes: on a 4xx (RFC 9309) a crawler may access any resource — a missing robots.txt is a free pass. On a 5xx (RFC 9309) it must assume a complete disallow.
- Specificity: the longest (RFC 9309) matching rule wins, measured in octets; on a tie,
Allowprevails. A broad block can be undone by a more specific allow — which may be intended, but should be a deliberate choice.
The 5xx boomerang
Google describes the link between server performance and crawl rate unambiguously: if a site responds quickly for a while, the limit goes up and more connections are used for crawling; if it slows down or responds with server errors, the limit goes down and Google crawls less (Google Search Central). Crawl rate is thus a function of your response time — and a slow site gets crawled less often without anyone having decided that. As candidates for crawl budget optimisation, Google names sites with one million (Google Search Central) or more unique pages, as well as sites with 10,000 (Google Search Central) or more pages that change daily.
In practice this means: faceted navigation, duplicate content, soft 404 pages and long redirect chains consume crawl budget without delivering anything in return (Google Search Central). Where parameter URLs are meant neither to rank nor to convert, they belong behind a robots.txt block. Permanently removed pages should return 404 or 410 rather than continue to be crawled as soft 404s (Google Search Central). The order matters: robots.txt prevents the fetch, not the indexing of an already known URL — conflating the two produces results nobody wanted.
Protecting the worker pool and the database
No server fails because of average load. It fails when a finite resource is exhausted. In PHP applications that is usually the worker pool: once the last free instance is taken, every further request waits in a queue — including the one from the customer who just clicked „place binding order“. TTFB then rises not linearly but in a jump. How that metric arises and where it tips over is described in the article on server response time.
- Size the worker pool against available memory rather than against a wish: more workers than RAM allows merely moves the problem into swap.
- Keep the database connection limit above the sum of all workers, so a full pool does not make the application fail at the database instead.
- Evaluate the slow query log specifically for facet and sort queries — that is where the most expensive access in the whole application happens.
- Set timeouts consistently: a request still computing after 30 seconds is blocking a worker that a paying customer needs.
- Route verified bots through their own upstream or pool, so their load stays capped and the purchase path keeps its own budget.
- Monitor saturation rather than CPU utilisation: occupied workers and queue length say more about the next outage than any utilisation graph.
Faceted queries are the most expensive guests here: they rarely hit an index cleanly, create temporary tables and scale poorly with the number of items. How such queries can be defused is shown in the article on database query optimisation. Whether the hardware fits the load profile at all is settled beforehand by the hosting choice — a model that looks fine when idle can break down considerably earlier under bot load.
Part of the load can additionally be reduced at the transfer volume. Where many similar HTML responses are delivered, delta compression noticeably reduces the amount transferred — the mechanics behind it are described in the article on shared dictionaries and delta compression. And for applications that navigate client-side, the measurement basis shifts anyway; the article on soft navigations in SPAs covers that.
The trade-off: visibility against capacity
The honest answer is this: a conflict of goals exists here, and it cannot be configured away. If you want to be visible in answer engines, you have to give crawlers access. If you give access, you pay with server capacity. Cloudflare quantifies the shift starkly: for every hour spent online searching for information, only 15 minutes (Cloudflare Radar) are spent on the open web. The rest of that usage ends in surfaces that read your content without anyone entering your site.
The decision is therefore not purely technical but commercial — and it is made per crawler, not across the board. A crawler with a ratio of 41 to 1 deserves different access than one at 38,066 to 1 (Cloudflare Radar). A training crawl across your filter combinations generally brings neither revenue nor visibility; a fetch that originates from a specific user question can bring both. That distinction is the real lever.
AI crawler load is thus mainly a dimensioning problem with a clear order: first measure what share of traffic is automated and which URLs it lands on; then shrink the crawl surface by taking facet and parameter URLs out of the path; then throttle and prioritise instead of banning; and only after that consider additional hardware. Reversing that order is expensive — a bigger server can be crawled empty faster, and little is solved by it. Getting the fundamentals of server optimisation and TTFB optimisation right creates room for both: content that stays reachable for machines and fast answers for customers.
Sources and studies