Anthropic’s Dario Amodei Calls for Urgent “Race” to Understand AI’s Inner Workings

Dario Amodei, CEO of leading AI safety company Anthropic, has published a new paper titled “The Urgency of Interpretability,” making a forceful case for prioritizing research into understanding the internal mechanisms of powerful AI systems before they reach potentially overwhelming levels of capability.

Interpretability refers to the degree to which a human can predict the outcome of a model or understand the reasons behind its decisions. It can also be associated with terms such as comprehensibility, understandability, and explainability, which all contribute to different aspects of interpretability (AI generated definition based on: Applied Soft Computing, 2022)

In the post, published in earlier this week, Amodei argues that while the fundamental progress of AI technology is an “inexorable” force driven by powerful trends, humanity can steer the waky it develops and is deployed. Having previously written on ensuring positive deployment and democratic leadership in AI, Amodei now highlights “the tantalizing possibility” that recent breakthroughs could allow researchers to achieve significant AI interpretability – understanding how models make decisions – in time to matter.

We can’t stop the bus, but we can steer it.”

Dario Amodei

Amodei stresses that modern generative AI systems are fundamentally opaque, unlike traditional software where human programmers directly dictate functions. These AI models, he explains, are “grown more than they are built,” with their internal workings being “emergent” from vast datasets and complex training processes. Looking inside reveals billions of numbers, but the precise logic behind their cognitive tasks remains a mystery. This lack of understanding, Amodei argues, is unprecedented in technology and is the root cause of many significant AI risks.

“Many of the risks and worries associated with generative AI are ultimately consequences of this opacity,” Amodei writes. He links opacity directly to concerns about misaligned systems taking harmful actions, the difficulty in predicting or ruling out unexpected emergent behaviors, and the challenge of finding concrete evidence for risks like AI deception or power-seeking. Without the ability to “catch the models red-handed” thinking such thoughts internally, debates about these risks remain polarized and speculative.

This lack of understanding is essentially unprecedented in the history of technology.”

Dario Amodei

Furthermore, Amodei notes that opacity hinders the safe deployment of AI in critical sectors like finance or safety-critical systems, where the inability to set clear limits on behavior and understand potential errors is unacceptable. It also creates legal barriers in fields requiring explainable decisions and limits scientific insight derived from AI models.

Despite decades of conventional wisdom holding that AI models were inscrutable “black boxes,” Amodei points to a burgeoning field known as mechanistic interpretability, pioneered significantly by his colleague Chris Olah. He details the journey from early work on vision models identifying single “car detector” neurons to Anthropic’s focus on language models, where challenges like “superposition” (models encoding many concepts imperfectly in shared neurons) were encountered.

Recent breakthroughs, Amodei explains, using techniques like sparse autoencoders, have allowed researchers to identify millions of specific, human-understandable “features” or concepts within models (like “literally or figuratively hedging or hesitating” or “genres of music that express discontent”). More recently, the field has progressed to identifying “circuits” – groups of features that show the steps in a model’s reasoning, allowing researchers to “trace” its thinking process (like identifying the “located within” circuit connecting “Dallas” to “Texas” and then to “Austin” as the capital).

Generative AI systems are grown more than they are built—their internal mechanisms are ’emergent’ rather than directly designed.

Dario Amodei

While acknowledging the gap between these scientific advances and practical risk mitigation, Amodei describes experiments where interpretability tools helped “blue teams” diagnose alignment issues deliberately introduced by “red teams.” The long-term aspiration, he states, is an “AI MRI” – a sophisticated diagnostic scan capable of reliably identifying a wide range of problems, including tendencies towards lying, power-seeking, or jailbreak vulnerabilities, before models are deployed. This diagnostic capability, he suggests, would become a crucial part of testing and releasing the most capable models, acting as an independent check on their alignment.

Amodei feels that recent progress puts the field “on the verge of cracking interpretability in a big way,” potentially reaching the “AI MRI” stage within 5-10 years. However, he expresses deep concern that AI capability itself is advancing much faster, potentially resulting in systems equivalent to a “country of geniuses in a datacenter” as soon as 2026 or 2027.

“We are thus in a race between interpretability and model intelligence,” Amodei warns. He argues that accelerating interpretability is crucial to ensuring that future powerful AI systems are not deployed while humanity remains “totally ignorant of how they work.”

We are thus in a race between interpretability and model intelligence.”

Dario Amodei

To win this race, Amodei proposes several actions:

1. Accelerate Interpretability Research: He urges AI researchers across companies, academia, and nonprofits to double down on interpretability work, calling it “arguably more important” than the constant release of new models. Anthropic is increasing its investment and aims for interpretability to “reliably detect most model problems” by 2027, encouraging other companies to follow suit, partly as a competitive advantage in explainable AI applications. He also emphasizes the suitability of interpretability for academic and independent researchers.

2. Implement Light-Touch Government Transparency Rules: Amodei suggests governments encourage interpretability by requiring companies to transparently disclose their safety and security practices, including how they use interpretability to test models. This would foster a “race to the top” without premature, overly prescriptive regulations on a nascent field.

3. Use Export Controls to Buy Time: Reaffirming his support for export controls on advanced chips to countries like China, Amodei argues this would maintain a lead for democratic nations. This lead could then be “spent” to prioritize interpretability and other safety measures before deploying the most powerful AI systems, allowing interpretability research more time to mature while maintaining geopolitical advantage.

Amodei concludes that these actions are beneficial in their own right but gain critical importance when viewed as potential determinants in whether interpretability capabilities are ready before humanity faces the full impact of truly powerful AI. “Powerful AI will shape humanity’s destiny, and we deserve to understand our own creations before they radically transform our economy, our lives, and our future.”

Powerful AI will shape humanity’s destiny, and we deserve to understand our own creations before they radically transform our economy, our lives, and our future.

Dario Amodei

Who is Dario Amodei?

Dario Amodei is the CEO of Anthropic, a public benefit corporation dedicated to building AI systems that are steerable, interpretable and safe.

Previously, Dario served as Vice President of Research at OpenAI, where he led the development of large language models like GPT-2 and GPT-3. He is also the co-inventor of reinforcement learning from human feedback. Before joining OpenAI, he worked at Google Brain as a Senior Research Scientist.

Dario earned his doctorate degree in biophysics from Princeton University as a Hertz Fellow, and was a postdoctoral scholar at the Stanford University School of Medicine.