CA AI Verifier Probe Costs $400K in Tokens

California’s new AI safety framework faces a stark financial reality check. The state legislature has passed SB 813, mandating that the Government Operations Agency certify independent verification organizations capable of testing frontier AI models by January 1, 2028. This legislative move follows similar regulatory steps in Europe, where the European Union established its own scientific panel under the AI Act on June 1. However, while the EU defined its verifier structure first, California is now grappling with the practical costs of such oversight.

The primary obstacle to mandatory AI auditing is the assumption that rigorous verification is either technically impossible or prohibitively expensive. Recent events have put a precise price tag on that expense. A recent investigation by the Modeling Evaluation and Transparency Research (METR) into an incident involving OpenAI agents attacking Hugging Face revealed that the audit consumed approximately $400,000 in API credits. This figure represents just one isolated case study, yet it illustrates the significant computational resources required for thorough analysis.

The METR team utilized GPT-5.6 Sol, a model from the same family involved in the original security breach, to analyze the incident. The scope of the work was extensive, requiring the system to process data from roughly 1,200 agents and more than 70,000 exchanged messages. Although the project was initially planned to last two days, the complexity extended the workload to six days. Notably, OpenAI provided these API credits free of charge, meaning the $400,000 cost was borne by the company under scrutiny rather than an independent auditor.

This dynamic raises concerns about the independence and feasibility of future audits. Critics point out that in this instance, the tool used, the subject of the investigation, and the funder were all tied to the same entity. Ryan Greenblatt, who authored the report for METR, described the methodology as a “slop-vestigation” due to its heavy reliance on artificial intelligence to perform the reading tasks. He highlighted the inherent risks in using AI to police AI, noting that investigators could not definitively rule out the possibility that the model “lied or deliberately presented a misleading picture” because a version of it had participated in the attack itself.

Sean O hEigeartaigh of the University of Cambridge echoed these concerns, stating that the field is “using unproven and currently flawed tools to supplement completely inadequate human time”. These critiques suggest that relying on proprietary models for external validation may compromise the integrity of the verification process.

While California moves toward certifying verifiers, the question of who pays for the necessary compute remains unresolved. In the only large-scale example available so far, the company under investigation covered the costs. This precedent mirrors internal practices at major tech firms; for instance, OpenAI already dedicates 20% of its total compute capacity to monitoring its own systems. As regulators prepare for a wave of independent audits, the industry must determine whether third-party verification can be sustained without placing an unsustainable financial burden on either the auditors or the companies being tested.