Cloudflare: Bot Traffic Could Hit 1,000x Humans by 2031

The most aggressive automated visitor to my site in the last 24 hours was a machine that claimed to be a well-known research crawler. It hit my server roughly 1,500 times, transferred nothing meaningful, and its primary objective appeared to be locating my SSH keys. That discovery sent me down a rabbit hole, prompted by a single, staggering number.

Cloudflare’s CFO, Thomas Seifert, told analysts on the company’s latest earnings call that if present growth patterns hold, non-human traffic could outpace human traffic by a factor of 1,000 within five years. His phrasing, which is already making the rounds, was stark: “humans will be a rounding error on the internet, not because human traffic goes down, but that’s just how fast we’re seeing non-human traffic grow.” Before the outrage machine kicks in, two clarifications are warranted. First, Seifert himself added a caveat, admitting he has “called it wrong at every point along the way.” Cloudflare had previously predicted machine traffic would surpass human traffic in 2027; it actually happened in May 2026. His misses have consistently been underestimates, which lends credibility to the forecast. Second, the underlying data is solid. Cloudflare’s own analysis from that same week confirms that fewer than half of all HTML page requests now originate from a human. That is a measured fact, not a marketing gimmick.

The real debate is what those numbers actually represent.

I decided to dig into the raw data for my own site, nohacks.co, pulling the AI crawler report for the 24 hours ending August 7. The result: roughly 3,000 requests, with about a third failing, a figure that jumped over 1,000% from the prior period. The breakdown by crawler was telling: CCBot led with 1,510 requests, followed by ChatGPT-User at 375, ClaudeBot at 296, Googlebot at 245, and PetalBot at 107. Thirteen other bots split the remaining 353.

CCBot is the crawler for Common Crawl, the long-standing nonprofit web archive whose dataset has trained a significant portion of the AI models currently under debate. On the surface, its status as my top visitor seemed unremarkable. Then I exported the actual request paths.

The most requested paths in that AI crawler traffic, with counts, looked like this: `/.ssh/knownhosts` (42), `/phpinfo.php` (31), `/.boto` (30), `/.env.production` (29), `/.vscode/launch.json` (28), `/.env.test` (27), `/firebase-service-account.json` (26), `/.gitconfig` (24), and `/server/.env` (24). The list continued for a hundred entries, including `/idrsa`, `/id_ecdsa`, `/private-key`, `/ssl/localhost.key`, `/.aws/config`, `/actuator/configprops`, and `/@fs/proc/self/environ`, the latter a known exploit attempt against a development server path-traversal bug. Across those hundred paths, there were 1,028 requests, 6.7 MB of data transferred, and zero referrals. Requests for anything I had actually written were virtually nonexistent. The closest it came to my content was a probe for `/blog/wp-login.php`, a WordPress login attack aimed at a site that has never run WordPress, and two hits on `/blog/null`.

That last detail is more significant than it appears. This traffic is not scanning my pages before asking for files. It is running a fixed list, the same list it runs everywhere, and my website is just another line in its loop. This is a credential scanner. Common Crawl, by design, follows links and fetches pages; it has no legitimate reason to ask a podcast site for its Firebase service account key. I cannot definitively prove impersonation from my end, as per-request IP data is unavailable on my plan. Common Crawl does publish a verification method: genuine CCBot traffic originates from documented address blocks and resolves to hostnames ending in `crawl.commoncrawl.org`. Anyone with those logs could settle it in a minute. What I can confirm is what arrived, what it asked for, and how it was labeled: Cloudflare’s AI dashboard attributes this traffic to Common Crawl as the operator and counts every request toward my AI crawler totals.

Here is what unsettles me most. I searched my security events for these requests and found nothing. The security log only records traffic that triggers a rule. Since I am not blocking this activity, it passes through, gets served, and leaves no trace. It appears in exactly one place on my dashboard: the AI crawler view, listed beside ChatGPT-User and Googlebot, under the banner of a nonprofit research archive. A credential scanner is fully visible to me as agent traffic and completely invisible as a security threat.

Two paths buried in that list are new, and they are the ones I keep circling back to: `/.mcp.json`, requested 30 times, and `/.continue/config.json`, requested 24. These are agent tooling configuration files: an MCP server definition and a coding assistant’s settings. Both routinely contain API keys and access tokens, because that is exactly what you store there to let an agent reach your services. Someone has added agent credentials to the standard secret-scanning wordlist. The same automated sweep that has been probing every website for `/.env` since forever now also asks for the file that lists which tools your agents can call and what they authenticate with. There was no announcement, and it happened quickly. If you run anything agentic, the wordlist arrived before most people finished writing their first MCP server.

The strongest counterweight to the earnings-call framing sits in Cloudflare’s own engineering blog from that same week. Their post on the agentic internet notes that a large share of traffic from well-behaved bots is re-fetching unchanged pages, running into the billions of requests. In their words, it is “an enormous amount of machine effort, attached to no outcome at all.” Machine effort and machine demand are not the same thing. My own logs prove that point more sharply than I expected: the largest single contributor to my machine traffic was not just useless, it was hostile, and it still counted.

Meta crawling your site and sending nothing back is the definition of useless traffic if you are the owner. A scanner wearing a research crawler’s name while hunting for your cloud credentials is a category below that, and both land in the same bar on the same chart. So when the graph climbs, the question for a website owner is what the traffic actually is.

It is worth being direct about what Cloudflare is doing here. Help create the problem, market the problem, sell the solution. In the first week of August alone, they delivered a bot-traffic projection on the earnings call, a blog post quantifying how much of the web is no longer human, an agent-readiness scanner to tell you that you are not ready, an AI-visibility product to score you, a bridge to expose your website’s tools to agents, and a default that starts blocking some of those agents in September unless you opt out. Every one of those products is a reasonable response to something real. That is precisely why the pattern is worth noticing rather than dismissing. The company measuring the problem, framing the problem, and selling the fix is one company, and they now own both the meter and the valve.

I want to be fair here, because I have supported much of what Cloudflare has done. Pay-per-crawl was the right call. Content Independence Day was the right call. Giving website owners a real choice over which machines get in beats a court deciding it for them, which is what I argued when the Ninth Circuit took up that question on August 4. All of that can be true at once. Cloudflare can do some good things, some directionally good things, and some things that look sketchy, simultaneously. Most companies can. The mistake is deciding they are the good guys or the bad guys and then reading everything they do through that lens.

Take the traffic numbers seriously, and take the framing with the salt it deserves. Machines are the majority of requests. That is measured, and it is true. Then open your own crawler analytics and read the paths, not the totals. Mine told me three things I did not know this morning: my largest AI crawler was a scanner, it was burning megabytes of my bandwidth on nothing, and the wordlist it works from now includes the config files it thinks my agent tooling lives in. None of that detail is in anybody’s projection. The volume is. Fifteen hundred of these arrived at one small website in a single day, every one of them counting toward the thousand-to-one ratio Seifert described to analysts, and not one of them wanted anything I wrote.