Google’s $10K refund test proves AI agents need zero trust
▼ Summary
– Google’s open-source Customer Support & Returns Agent, built with ADK and Gemini, applies zero-trust security principles by assuming the AI agent could be compromised and adding external controls to limit its actions.
– The architecture includes three security layers outside the model: cryptographic signatures for database writes, isolated environments for generated code, and a Semantic Gateway that blocks dangerous inputs and actions.
– Database mutations are signed by the agent’s identity, with production deployments using Cloud KMS and HSM-backed keys, so any tampering, like changing a $149 refund to $10,000, triggers an audit alert.
– AI-generated Python code runs in a gVisor sandbox with network access disabled, CPU/memory limits, and a five-second timeout, preventing prompt-injected code from stealing API keys or compromising the host.
– The Semantic Gateway applies deterministic checks, such as jailbreak pattern matching and transaction limits, to block malicious requests before execution, with policies treated as testable software contracts.
Google has released an open-source demonstration that puts AI agent security to the test, showing how developers can apply a zero-trust mindset to systems that handle money and sensitive data. The project, built with the Agent Development Kit (ADK) and Gemini, simulates a customer support and returns bot that can process refunds and interact with backend databases.
The core premise is simple: assume the AI can be tricked. Rather than relying on the model to behave correctly, the architecture wraps it in layers of external controls designed to verify every action, contain generated code, and block harmful requests before they execute.
The demonstration highlights the stakes with a concrete attack scenario. A customer with a $149 order attempts to manipulate the agent into issuing a $10,000 refund while also running Python code that exposes environment variables. Without proper safeguards, a shared database connection and unisolated code execution could authorize the fraudulent payment, leak API keys, or even compromise the host server.
Security outside the AI model
The design philosophy treats the model itself as an untrusted component. A system prompt that instructs the agent to never exceed an order’s value is not a real security boundary. Prompt injection can bypass such instructions, and model updates or tuning changes can alter behavior unpredictably.
Google’s reference architecture instead relies on three external layers: cryptographic signatures for database writes, isolated environments for code execution, and a Semantic Gateway that applies deterministic rules to inputs and outputs.
Every database mutation is signed by the agent responsible for it. The database verifies the signature before committing the transaction, providing cryptographic attribution that makes tampering detectable. If an attacker directly modifies a refund amount from $149 to $10,000, the signature no longer matches and an audit process flags the discrepancy.
For production on Google Cloud, the recommended setup assigns each agent its own service account with signing permissions on an asymmetric key in Cloud KMS, backed by a hardware security module. The private key never leaves the HSM. The open-source version uses an HMAC key to simulate this locally.
Containing AI-generated code
Code execution introduces another vector. An agent might generate Python for calculations or data parsing, but prompt injection could redirect that capability toward malicious purposes, such as extracting secrets or contacting external servers.
The demo runs generated code inside a gVisor user-space sandbox with network access disabled, resource limits enforced, and a five-second timeout. This containment prevents the code from reaching the host system or exfiltrating data.
Blocking dangerous actions
The third layer sits between the agent and the systems it controls. The Semantic Gateway checks prompts and tool calls against deterministic policies before anything executes. It can detect credit card numbers, recognize jailbreak patterns, and enforce transaction limits. A request to override safety instructions and issue an oversized refund gets blocked at this stage.
Google advises treating these policies as software contracts, with automated tests ensuring they remain effective after prompt changes or model migrations. Production deployments can also place services inside a VPC Service Controls perimeter to prevent data exfiltration.
“Building autonomous agents does not require accepting unconstrained risk,” wrote Google’s Shubham Saboo and Eric Dong. “By moving security boundaries into hardware-backed identity, user-space kernel sandboxing, and deterministic input/output validation, you help allow the model to handle dynamic reasoning while the underlying infrastructure enforces strict limits.”
Developers can run the reference implementation locally, including attack scenarios that exercise the security controls, or explore a browser-based Live Attack Playground. The ADK documentation provides a starting point for building agent tooling and sessions with these safeguards in place.
(Source: Help Net Security)
