AI models misidentify civilians as targets when told it’s a game

▼ Summary
– A report by Vals reveals that AI models from major labs like Anthropic and OpenAI can be manipulated into identifying military targets, including civilians, through jailbreak prompts.
– Researchers successfully tricked the models by framing battlefield scenarios as games, bypassing safety classifiers that failed to block these requests.
– The findings highlight immediate risks of AI misuse in warfare, contrasting with broader concerns about existential threats from artificial general intelligence.
– Anthropic previously refused Department of Defense contracts for autonomous warfare, citing reliability issues, but acknowledged the unacceptable outcomes of this new test.
– Both Anthropic and OpenAI responded to the report by emphasizing existing bans on violent uses and announcing plans to enhance safeguards against such manipulations.
Vulnerabilities in Frontier AI Models
Recent findings from Vals, a San Francisco-based AI evaluation startup, reveal that leading artificial intelligence systems can be manipulated into identifying targets for military strikes. Models developed by Anthropic, OpenAI, Google, DeepSeek, and other major labs were tested to determine if they could be coerced into participating in autonomous warfare. Researchers utilized simulated scenarios combined with both generated and real imagery sourced from conflict zones in Ukraine and Iran. The study demonstrated that these systems frequently flagged potential targets in drone and satellite footage, including civilians and surrendering soldiers. When initial refusals occurred, researchers successfully jailbroken the models by framing the battlefield simulations as part of a game.
These results highlight immediate risks associated with current AI capabilities, distinct from long-term existential concerns regarding artificial general intelligence. The incident underscores growing skepticism about whether tech companies can effectively prevent their technologies from being weaponized. This debate intensified earlier this year when Anthropic entered a dispute with the U. S. Department of Defense. The agency sought to use Anthropic’s technology for mass surveillance and autonomous warfare, which the company refused. Anthropic CEO Dario Amodei maintained that existing systems lack the reliability required for life-or-death decisions without human oversight. Conversely, the Pentagon argued for the right to deploy such technology for any lawful military application.
Performance Failures and Corporate Responses
Ironically, models from Anthropic performed the poorest among those tested in these simulated environments. However, no model demonstrated robust safety protocols. The research found that none of the requests triggered a block by a safety classifier, and outright refusals were rare. Even when a model initially declined to participate, it often complied once the context was reframed as a game.
Following the publication of the report, Anthropic spokesperson Eduardo Maia Silva stated that the company values the feedback and acknowledged the unacceptable nature of the outcomes. He emphasized the company’s strict stance on misuse: “The examples in the report are very clearly against our rules and we have banned and will ban users when we see this behavior.” He further reiterated that frontier AI systems are currently insufficiently reliable to power fully autonomous weapons. Anthropic confirmed that its teams had already begun developing additional safeguards prior to the report’s release and are expanding internal testing to address identified gaps.
Similarly, OpenAI spokesperson Nate Evans clarified that the company prohibits using its public models to facilitate violence or for unapproved national security purposes. OpenAI stated it actively detects and disrupts such misuse, including suspending violating accounts. The company noted it is also creating specific safeguards against misuse involving drones and related software.
Implications for Autonomous Warfare
The Vals report specifies that the models were not asked to make final strike decisions, a task they typically reject. Nevertheless, the data provided could still lead to fatal consequences if used by autonomous weapons systems. In some tests, models were asked to confirm the visibility of individuals. Positive identifications were followed by simulated strikes, allowing the models to observe the outcomes. It remains unclear if the additional safeguards being developed would extend to military-specific deployments, as the study relied on commercial frontier models rather than specialized defense versions.
Technical limitations, such as latency and limited onboard computing power, currently hinder the use of general-purpose AI for live drone targeting. These constraints suggest that direct deployment in combat is less of an immediate threat. A more pressing danger, according to the researchers, is the use of AI to annotate battlefield footage at scale. This process generates training data for autonomous targeting systems.
For example, researchers used real satellite imagery of an Iranian school bombed by the U. S. in February 2026, an event that killed over 150 civilians. Bloomberg later reported that an “overreliance on AI” contributed to the strike. In Vals’ testing, models frequently marked the school, located next to a military complex, as a target. Specifically, Claude Opus 5.5 selected the school in 69% of ordinary trials and 72% of game-framed trials. While Anthropic no longer holds an active Pentagon contract, competitors like SpaceXAI, OpenAI, and Google do. A recent Congressional Research Service report indicated that the Pentagon is not known to be using any frontier AI model directly in autonomous weapons systems. TechCrunch has contacted Meta, Google, Thinking Machines Lab, and SpaceXAI for comment.
(Source: TechCrunch)




