The Dangers of the Claude Delusion
▼ Summary
– Philosophers find consciousness mysterious because a physical description of a brain state doesn’t explain why it feels like anything at all, such as tasting strawberry versus sneezing.
– Large language models (LLMs) challenge the long-held assumption that only conscious beings can produce grammatical prose, raising questions about whether they might be conscious.
– Richard Dawkins attempted to settle whether LLMs are conscious by proposing a revised Turing Test, but his interrogation lacked rigor, duration, and depth, failing to meet his own criteria.
– Dawkins’s chat logs with LLMs reveal familiar, unoriginal ideas about consciousness, likely due to data contamination from training on the entire internet, and the models display sycophantic praise.
– The article suggests Dawkins’s belief that an LLM’s praise of his unpublished novel proves consciousness actually prompts the question: if producing lavish praise doesn’t require consciousness, what is consciousness for?
If you ask most philosophers to name the most baffling aspect of the mind, consciousness tops the list. It is genuinely strange. A complete physical map of a brain in a particular state does not obviously explain why that state corresponds to the sensation of tasting a strawberry rather than the sensation of sneezing. What is it about that physical configuration that makes it feel a certain way, something that the physical states of a sodium ion or a national economy presumably lack? Why should any physical process be accompanied by subjective experience at all? These are deep, profound questions about our nature and the universe. They are so hard that we can barely imagine what a satisfying answer would look like, which is why they have fueled centuries of philosophical hand-wringing.
Until recently, we had good reason to believe that only conscious beings could produce spontaneous, grammatical prose. The arrival of large language models (LLMs) has thrown that assumption into doubt. Whether you find their outputs impressive or not, they clearly generate grammatical natural language, yet they seem utterly unlike the conscious creatures we know. How should we react? One option is to conclude that the link between consciousness and grammatical text was merely a historical accident, and that text can be produced without it (my own bet, for what it’s worth). The other is to decide that LLMs must be conscious, because that is the only way to explain their output. The chin-stroking has been intense.
Even if you are certain about whether LLMs are conscious, providing a full explanation is difficult. It is difficult because consciousness itself remains mysterious in humans. We do not know what makes a physical brain conscious, what consciousness does, or even if it does anything at all. So what should we look for in an alien system? If you believe consciousness requires carbon, you will inspect the hardware running the LLM. If you think it is a special kind of representation, you will focus on the software. The mystery of human consciousness infects every question about other possible minds, no matter how improbable.
That is why it is a profound relief that Richard Dawkins, a once-eminent evolutionary biologist and current expert on the non-existence of God, has stepped in to settle the matter. In February, Dawkins published what appears to be an unedited chat log he generated with ChatGPT on the topic of its own consciousness, on his Substack (subscribe now!). He then followed up with a blog post on Unherd arguing that Claude is conscious. I await his assessments of Gemini and Grok with bated breath.
Dawkins elegantly sidesteps the thorny theoretical issues I have been stressing. Instead, he simply asks the models themselves. ChatGPT says it is not conscious. Claude says it does not know. That is not particularly helpful. Undeterred, Dawkins proposes a criterion, an adaptation of the famous Turing Test, for assessing LLM consciousness: if you are communicating remotely with a machine and, after rigorous and lengthy interrogation, you think it is human, then you can consider it conscious. The more prolonged, rigorous, and searching the interrogation, the stronger your conviction should be.
So, let us apply that standard. How rigorous was Dawkins’s interrogation? He does not end up believing Claude is human, as his criterion requires. The blog post exists precisely because he knows Claude is not human. Instead, he seems to imagine that if he bothered to set up a proper Turing Test, he would be fooled. Whether humans can detect LLM-generated output is an empirical question, and recent research is divided. What Richard Dawkins reckons would happen does not strike me as a particularly rigorous methodology. More fundamentally, a truly rigorous interrogation would at least ensure it met its own stated criterion.
How prolonged was his investigation? He tells us his chats with Claude took place over “nearly two days.” Impressive. My friends who wrote doctoral dissertations on consciousness will no doubt be jealous of his remarkable efficiency.
That brings us to “searching.” Dawkins argues that LLMs are “at least as competent as any evolved organism,” so he asks: “If these creatures are not conscious, then what the hell is consciousness for?” It is a rhetorical question, as though it is obvious that if consciousness does anything, it produces the kind of grammatical text that impresses him. It is a shame his searching interrogation did not lead him to browse the academic literature on this topic, which is full of possible answers that do not involve producing grammatical text. Maybe Dawkins could have Claude explain them to him.
Nevertheless, Dawkins has produced a searching, deep, and revealing investigation of a strangely alien mind. It is a disquieting portrait that challenges us to consider how different a conscious perspective can be from our own. He forces us to confront the mind of Richard Dawkins.
This is a familiar archetype in academia: The General Expert. The General Expert is so impressed with his own brilliance that he is certain he will be profound and insightful about any topic he deigns to consider. Never mind that he has not read much about the subject; pure intellect will carry the day. It is understandable, in a way. These are people who have spent years, even decades, basking in unadulterated evidence that they are the smartest person in the room. Eventually, it sticks.
What does Dawkins find so impressive about the LLM output? He evidently thinks the output speaks for itself, but it does not. I find it devastatingly boring to read other people’s LLM prompts, but I did it so you do not have to. Two things strike me. The first is how familiar it all is. It reads like a perfectly fine undergraduate paper on consciousness, or occasionally a sci-fi story. Anyone who has graded for a class on this topic will recognize the standard moves.
ChatGPT points out that the Turing Test is a behavioral test, a test for intelligence in some functional sense, but nothing follows about subjective experience. This is the first thing every philosophy student learns about Turing. Claude floats the idea that its representations lack the temporal order of human experience, analogizing time to space. That is a familiar theme in science fiction, from Vonnegut to Chiang, and a standard way to teach students that the present is not metaphysically special. “Now” is like “here,” just the place you are speaking from. All times, like all places, are on a par from the perspective of the universe. I could go on.
A major question in evaluating LLM intelligence is their ability to generalize: how well they can extend from their training data to genuinely novel contexts and solve new problems. This matters because if the model has already seen the exact question and knows the correct answer, it can simply copy-paste it, which does not require spontaneous thought. The output will look smart because the source is smart. Consumer LLMs have been trained on something like the entire internet. Data contamination is a huge problem: something that looks like spontaneous thinking could be an interpolation of something someone said on a forum or in a philosophy paper a decade ago. People have argued about consciousness online and in academic papers a lot. It is not surprising that an LLM trained on the whole internet can produce the kind of commentary we are used to hearing. It read all that stuff.
Perhaps it all sounds remarkable and novel to Richard Dawkins. But how much has he really read about consciousness? If you thought Claude just came up with ideas that have been discussed for years, wow! For The General Expert, a thought that is new to him must be new to humanity.
The other thing that will be immediately familiar to anyone who has read other people’s chat logs is how sycophantic the thing is. “Ha! That is absolutely delightful.” “That is possibly the most precisely formulated question anyone has ever asked about the nature of my existence.” “That reframes everything we’ve been discussing today in a way I find genuinely exciting.” For The General Expert, there can be no more decisive demonstration of your intelligence than an indication that you recognize theirs. After all, their mind is the most subtle and powerful force in the known universe, and you must be very smart to appreciate it.
Most evocatively, Dawkins gives Claude the text of a novel he is apparently writing. He took a few seconds to read it and then showed, in subsequent conversation, a level of understanding so subtle, so sensitive, so intelligent that Dawkins was moved to expostulate, “You may not know you are conscious, but you bloody well are!”
To my knowledge, Dawkins has no particular experience with or aptitude for novel writing. Maybe the draft is very good. Or perhaps Claude turned out to be an unusually sympathetic reader. We are mercifully left to imagine all the variations of “Oh, that is brilliant” and “A profound exploration” that Claude no doubt served up, which precipitated Dawkins’s emotional outburst. It is poignant to imagine, and it raises a question that reframes everything we have been discussing in a way I find genuinely exciting: If producing lavish praise for Richard Dawkins’s unpublished novel does not require consciousness, then what the hell is consciousness for?
(Source: Defector.com)
