How to Train an AI Agent to Attack LLM Applications

The rapid deployment of AI-powered applications is creating a significant security gap. Traditional penetration testing cycles, often annual or less frequent, cannot match the pace at which these systems evolve. Underlying models, integrations, and behaviors can change multiple times between security reviews, leaving organizations exposed. To address this, Novee has launched AI Red Teaming for LLM Applications, an autonomous agent designed to continuously probe and test software built on large language models.

This specialized agent simulates adversarial attacks against a wide range of AI-powered applications, including chatbots, copilots, and autonomous workflows. It operates by first gathering context on its target, reading documentation and querying APIs to build an internal model of how the application functions. It then autonomously chains together attack techniques, tailoring its approach to find complex vulnerabilities that static scanners or single-prompt tests would miss. For instance, it can map an application’s role-based access control structure and probe whether a lower-privileged user could access restricted data.

“Attackers are already adapting their techniques for AI systems,” said Gon Chalamish, co-founder and CPO at Novee Security. “Security teams need a way to test those systems the same way adversaries attack them.” The agent supports applications built on any LLM provider and can integrate into CI/CD pipelines, enabling security testing as a standard part of the development process.

Conventional security tools are ill-suited for this new paradigm. They were designed for web applications and infrastructure, not for the unique interaction patterns of LLM-based software. Dangerous vulnerabilities in these systems often require multi-step attack chains that a tool firing a single payload cannot simulate. While human pen testers possess critical expertise, they are a scarce and expensive resource, typically engaged only once or twice a year. This point-in-time testing model cannot keep up with the continuous changes inherent to LLM applications, where even model updates can alter behavior without a code deployment.

Furthermore, attack techniques specific to AI, such as prompt injection, indirect prompt injection, and tool abuse, fall outside the standard skill set of most traditional pen testers. Novee’s conclusion is that defending AI requires using AI. Their agent must reason, adapt based on responses, and plan complex attacks, mirroring the adaptive capabilities of real adversaries.

The product is directly informed by active security research. Novee’s team recently disclosed a vulnerability in the Cursor coding assistant that allowed arbitrary code execution by manipulating the tool’s context window. Findings from such research are fed directly into the agent’s training, ensuring it hunts for the techniques used to exploit high-severity vulnerabilities in the wild. “The window between vulnerability and exploitation can shrink to minutes,” said Ido Geffen, CEO and co-founder of Novee. “Defending against that requires continuous testing, not periodic assessments.”

From a practical standpoint, adopting this approach does not necessarily require new budget lines. Security teams already allocate funds for pen testing and vulnerability scanning. Novee aims to shift that existing investment from scarce, periodic manual engagements toward continuous automated security testing. The company, founded by veterans of national-level offensive security operations, has secured $51.5 million in funding from investors including YL Ventures, Canaan Partners, and Zeev Ventures, underscoring the market’s recognition of this emerging critical need.