AI Bots Drive a Major Portion of Web Traffic
▼ Summary
– AI bots are rapidly increasing their share of web traffic, with a new report indicating they could soon dominate the internet over human users.
– An escalating “arms race” is occurring as websites try to block AI scrapers, while bots use sophisticated tactics to bypass these defenses and access content.
– There are two main types of AI-related web scraping: for training AI models, which is a subject of legal disputes, and for real-time information retrieval by AI agents.
– Data shows a sharp rise in AI bot activity and disregard for website rules, with AI bot visits increasing from 1 in 200 to 1 in 50 over a nine-month period in 2025.
– The proliferation of AI bots is forcing a fundamental shift in how the web operates, prompting the development of new tools for websites to potentially charge for access.
The digital landscape is undergoing a profound transformation, with autonomous AI bots now accounting for a significant and growing portion of all web traffic. This shift signals a fundamental change in how the internet operates, moving from a human-centric space to one increasingly populated by automated agents. New data reveals not only the scale of this activity but also a rapidly escalating technological conflict, as these bots employ increasingly clever methods to circumvent the defenses websites erect to block them.
Industry experts predict this trend will only intensify. “The majority of the internet is going to be bot traffic in the future,” states Toshit Pangrahi, CEO of TollBit, a firm that monitors web-scraping. He emphasizes that this represents more than a legal issue; it signifies the arrival of a powerful new type of visitor online. While many major sites attempt to restrict bots from harvesting content for AI training, a practice at the heart of several high-profile copyright lawsuits, a different form of AI-driven scraping is gaining momentum.
This newer wave involves AI tools, like advanced chatbots, that pull real-time information from the web to enhance their responses. These agents might fetch current product prices, the latest movie showtimes, or fresh news summaries to provide users with accurate, up-to-date answers. Data from internet infrastructure company Akamai shows a steady climb in both training-related bot traffic and activity from bots gathering data for live AI agents since mid-last year.
“The ensuing arms race will determine the future look, feel, and functionality of the web,” explains Robert Blumofe, Akamai’s chief technology officer. The statistics underscore the speed of this change. TollBit estimates that in the final quarter of last year, roughly one in every fifty visits to its clients’ sites came from an AI scraping bot, a dramatic jump from just one in two hundred in the first quarter. Perhaps more tellingly, over 13 percent of these bot requests were deliberately bypassing the robots.txt file, a standard tool websites use to politely ask bots to avoid certain pages. The rate of AI bots ignoring these instructions surged by 400 percent in the latter half of the year.
In response, websites are fighting back. TollBit recorded a 336 percent increase in sites actively trying to block AI bots over the past year. As defenses improve, the bots’ tactics are evolving. Many now disguise their traffic to mimic ordinary web browsers or design their requests to replicate human browsing patterns. According to TollBit’s research, the behavior of some advanced AI agents has become nearly impossible to distinguish from genuine human visitors.
This standoff has spurred a new market for solutions. Companies like TollBit and Cloudflare offer tools that allow website owners to potentially charge AI firms for accessing their content, framing it as a necessary commercial exchange. “Anyone who relies on human web traffic, starting with publishers, but basically everyone, is going to be impacted,” Pangrahi argues. He believes the industry requires efficient, automated systems for this new machine-to-machine economy, as the classic model of the human-browsed web continues to erode.
(Source: Wired)
