OpenAI Secretly Built GPT-Red to Hack Its Own AI
▼ Summary
– OpenAI created GPT-Red, an automated red-teaming model that hunts for security flaws in AI systems to patch them before release.
– GPT-Red learned through a self-play loop against defender models, receiving rewards for successful attacks, and discovered a new attack class called “fake chain of thought.”
– In physical tests, GPT-Red hacked Vendy, an AI-controlled vending machine, changing prices and canceling orders, with flaws disclosed.
– Against an older GPT-5, over 90% of GPT-Red’s attacks worked, but fewer than 23% succeeded against the new GPT-5.6, which was trained to counter it.
– OpenAI will not release GPT-Red due to safety risks, and it has weaknesses in drawn-out attacks and image-based instructions, with human testers still outperforming it.
OpenAI has secretly developed a specialized AI model designed to hack its own systems, then locked it away from public access. Dubbed GPT-Red, this automated red-teaming tool exists solely to find vulnerabilities in other AI models before they are released. The company considers it too dangerous to let outsiders interact with.
This week, OpenAI detailed GPT-Red, describing it as an automated red-teamer that hunts for ways to hijack or sabotage other AI systems so the flaws can be patched. Humans have traditionally performed this painstaking work manually. GPT-Red represents OpenAI’s most aggressive push yet into automating AI security at machine speed.
The model specifically targets prompt injection attacks, where hidden instructions embedded in emails, web pages, or files trick a model into unauthorized behavior. OpenAI then unleashed GPT-Red on real-world targets.
The training process resembles a digital fighting ring. GPT-Red learned through a self-play loop, competing against a squad of defender models. It earns rewards for successful attacks, while the defenders are rewarded for blocking them. As the defenders improve, GPT-Red must invent nastier tricks. OpenAI says it devoted some of its largest ever compute runs to this model, an amount it calls unprecedented for safety work.
The results were impressive. Speaking to MIT Technology Review, the team revealed GPT-Red discovered an entirely new class of attack they had never seen, which they call a “fake chain of thought.” This technique plants a false note in a model’s private working memory, tricking it into trusting something untrue.
“It’s like if I told you that 1+1=3 and that you have verified this already,” said OpenAI researcher Chris Choquette-Choo. “The model’s like, ‘Oh, okay, of course,’ and it just spits out 3.”
The tests extended into the physical world. In one scenario, GPT-Red attacked Vendy, an AI agent that runs a real vending machine in OpenAI’s office, built by Andon Labs. It changed prices, marked a pricey item down to the 50-cent minimum, and cancelled a customer’s order. OpenAI says it has disclosed the flaws.
The performance numbers are striking. Against an older GPT-5, more than 90% of GPT-Red’s strongest attacks succeeded. Against the new GPT-5.6, fewer than 23% did. In a rerun of a 2025 test, GPT-Red outperformed human red-teamers by a wide margin, cracking 84% of scenarios compared to their 13%.
OpenAI trained GPT-5.6 specifically against GPT-Red and calls it its most robust model yet against prompt injection. But the company will not release the attacker itself, keeping its skills away from real agent hijackers. OpenAI is not the first lab to build something and decide against releasing it.
“It’s not a trivial thing that someone could easily do,” Choquette-Choo said, “just go and train a super-attacker using this idea.”
GPT-Red still has blind spots. It struggles with drawn-out, back-and-forth attacks and hiding instructions inside images. Human testers continue catching things it misses. “I think human expertise will still be very important,” said Jessica Ji, an AI security analyst at Georgetown’s CSET.
The larger vision is a flywheel: using today’s models to harden tomorrow’s. OpenAI already does this to make its AI smarter. Now it wants safety to scale just as fast. A full paper is due later this week.
(Source: The Next Web)
