The Internet’s Next Era: Built for Machines
▼ Summary
– AWS launched a new generation of OpenSearch Serverless, a managed search and vector database designed specifically for AI agent workloads that spike unpredictably and then go idle.
– The new system decouples compute from storage, allowing compute to scale up in seconds for agent traffic bursts and down to zero to avoid charging customers for idle capacity.
– Cloudflare reports that bots accounted for 31% of HTTP traffic over the last six months, and predicts non-human traffic will exceed human traffic by the first half of 2027.
– The launch reflects a broader industry shift, with companies like Databricks, Snowflake, Microsoft, and Cloudflare redesigning infrastructure to handle machine-generated traffic from AI agents.
– OpenSearch Serverless integrates natively with AI development platforms like Vercel and Kiro, enabling developers to deploy search backends for agents without managing infrastructure.
Cloud infrastructure was originally architected for human behavior: predictable browsing, methodical searching, and steady streaming. AI agents operate on a completely different rhythm. They can erupt in bursts of activity, spawning multiple sub-agents that simultaneously query hundreds of databases, scan documents, and call APIs within seconds, then vanish just as abruptly.
Operating on this premise, Amazon is overhauling a fundamental component of its cloud infrastructure. On Thursday, AWS introduced its next-generation OpenSearch Serverless, a fully managed search and vector database designed for large-scale information storage and retrieval, now tailored specifically for agentic workloads. According to AWS, the system can instantly scale up when agents initiate tasks and scale back to zero during idle periods.
This launch signals a broader industry recognition: Infrastructure built for a human-driven internet is ill-suited for an agent-dominated world.
Although AI agents still account for a modest share of internet activity, machine-generated traffic is already substantial and growing rapidly. Cloudflare reports that bots comprised 31% of overall HTTP traffic over the past six months, with AI crawlers, search engines, and assistants representing roughly a quarter of all bot requests during that span.
“Non-human traffic will surpass human traffic sometime in the first half of 2027,” predicted Lai Yi Ohlsen, senior product manager at Cloudflare, in an interview with TechCrunch.
At Google’s I/O developer conference last week, the company announced that users will soon delegate tasks to AI systems, including researching purchases, booking travel, browsing the web, and interacting with apps. But the shift extends beyond consumer agents. Enterprises are increasingly deploying agents internally and for customers, generating new forms of machine-to-machine traffic behind the scenes.
Consequently, cloud providers and infrastructure firms are grappling with how to adapt human-centric systems to agents that constantly and autonomously retrieve information, invoke tools, and generate traffic. AWS’s new OpenSearch Serverless directly addresses this challenge.
“The timing is straightforward. Agents are moving from experimentation into production, and they create traffic patterns that previous infrastructure simply wasn’t designed for,” said Tia White, general manager for Amazon OpenSearch Service, to TechCrunch. “They spike without warning, they go idle without notice, and enterprise needs search that keeps up without paying for empty or idle compute.”
The key technical innovation in this generation is the decoupling of compute from storage. This allows compute resources to scale up in seconds to accommodate agent traffic bursts and scale down to zero, meaning customers pay nothing when agents are idle.
“Previously, even in our prior Serverless version, you had to have at least one instance operational and running because storage and compute were coupled,” White explained. “You couldn’t just automatically spin up [compute] at the rate you needed to, so you always had idle compute reserved for your workload, whether you were using it or not.”
Think of it as always paying for a parking space, even when empty. With AWS’s upgraded Serverless, it’s more like paying for a metered spot.
At launch, OpenSearch Serverless integrates natively with AI development platforms like Vercel and Kiro, enabling developers to deploy production-ready search and vector backends for agents without managing infrastructure.
This shift is rippling across the cloud industry. Databricks and Snowflake are repositioning themselves as AI memory and retrieval systems for enterprise data. Microsoft has rolled out Azure updates designed to handle AI agent bursts and share memory between agents. Cloudflare, similarly, introduced infrastructure last month aimed at giving agents persistent environments and instant scalability.
As more companies deploy AI agents, the pressure to redesign infrastructure around machine-generated workloads will intensify. This, in turn, could make agents cheaper and easier to deploy at larger scales, accelerating the transition to an internet built for machines.
(Source: TechCrunch)
