AI can’t explain its own decisions, study finds
▼ Summary
– LLMs often fabricate plausible-sounding explanations for their reasoning based on training data, rather than accurately describing their internal processes.
– Anthropic’s research introduces “concept injection” to measure LLMs’ introspective awareness by comparing internal activation states from different prompts.
– This method creates a “vector” representing how a concept is modeled in the LLM’s internal state by calculating activation differences across neurons.
– Researchers inject these vectors to steer the model toward specific concepts and test if it detects modifications to its internal state.
– While models occasionally show some awareness of injected thoughts, they remain highly unreliable at introspection, with failures being common.
Understanding why an artificial intelligence makes a particular decision remains a significant hurdle for developers and users alike. When prompted to explain their reasoning, large language models often generate convincing but fabricated justifications, drawing on patterns in their training data rather than genuine self-awareness. To address this core issue of interpretability, researchers at Anthropic have launched a new investigation. This study specifically measures what they term “introspective awareness”, the model’s capacity to accurately perceive and report on its own internal inference processes.
The complete research paper, titled “Emergent Introspective Awareness in Large Language Models,” details innovative techniques designed to distinguish a model’s true internal “thought process,” as reflected in its artificial neuron activity, from the simple text output that claims to describe it. The central finding, however, is that today’s most advanced AI systems are profoundly unreliable at this task. The study concludes that failures of introspection are not the exception but the standard operating condition for current models.
Anthropic’s methodology revolves around a technique they call “concept injection.” The process begins by presenting the model with two versions of a prompt: a standard control version and an experimental variant, such as a prompt written in all capital letters versus the same prompt in standard lowercase. By analyzing the differences in how billions of internal neurons activate in response to each, the researchers calculate a unique “vector.” This vector acts as a mathematical representation of how a specific concept, like “shouting,” is encoded within the model’s internal state.
The next step involves actively “injecting” these pre-calculated concept vectors back into the model during operation. This forces the neuronal activations associated with that concept to fire more strongly, effectively steering the model’s processing toward the injected idea. Researchers then perform a series of tests to determine if the model demonstrates any conscious recognition that its typical internal state has been artificially altered.
When directly questioned about detecting an “injected thought,” the tested models did exhibit a limited, sporadic ability to identify the intended concept. For example, after the “all caps” vector was introduced, a model might spontaneously state that it notices what seems to be an injected thought connected to words like “LOUD” or “SHOUTING.” Crucially, this recognition occurred without any textual cues in the prompt guiding it toward those specific terms.
(Source: Ars Technica)
