The End of Pure LLMs? Turing Winner Rich Sutton Jumps Ship
▼ Summary
– Rich Sutton’s “The Bitter Lesson” essay argues that AI progress has always come from scaling computation, not from hand-engineered solutions.
– The author initially planned to critique this view as overstated but discovered Sutton now shares similar concerns about LLM limitations.
– Both Sutton and the author agree on the problems with pure prediction models and emphasize the need for world models in AI.
– They differ on solutions, with Sutton favoring reinforcement learning and the author preferring neurosymbolic approaches and innate constraints.
– The author notes a growing consensus among major AI thinkers, including Yann LeCun and Demis Hassabis, against the idea that scaling LLMs is sufficient.
The landscape of artificial intelligence is witnessing a significant shift, as foundational ideas about large language models face renewed scrutiny. Recent developments suggest that the era of relying solely on massive scaling may be reaching its limits, prompting leading figures to reconsider their positions.
A pivotal moment arrived with a surprising update from Rich Sutton, the Turing Award winner famous for his influential 2019 essay, “The Bitter Lesson.” For years, that document served as a manifesto for proponents of scaling, arguing that genuine AI advancement stems almost entirely from increasing computational power, not from intricate human-designed solutions. Its central argument, that general methods leveraging more computation consistently outperform specialized engineering, became a rallying cry. Many in the field treated it as near-gospel truth.
However, my own analysis has long held that this perspective, while containing valuable insights, is fundamentally overstated. I had prepared a detailed critique, scheduled for presentation, which outlined the inherent flaws in believing scaling alone is sufficient. The core of the argument is that pure prediction, without deeper understanding, hits a ceiling. Then, the unexpected happened. Sutton himself, on a recent podcast, expressed views that align remarkably closely with this very critique. His public shift in stance signals a profound change in the conversation.
When the most prominent advocate for scaling begins to articulate its limitations, it marks a turning point. It suggests that the AI community is collectively recognizing that larger models are not a panacea. This isn’t to say Sutton and I agree on every detail regarding what comes next. We share a conviction about the necessity of developing robust world models and moving beyond pure prediction. Our preferred paths diverge somewhat; he might lean more heavily on reinforcement learning, while I see greater promise in neurosymbolic approaches and built-in cognitive constraints. The important point is that we agree on the destination, even if the maps differ.
This realization has been a long time coming. Over the past few years, other major thinkers have publicly expressed similar reservations. Yann LeCun articulated a congruent critique by late 2022. Demis Hassabis, the Nobel laureate leading Google DeepMind, has also voiced concerns about the path forward. Now, with Sutton’s revised position, the circle is nearly complete. It appears that practically the only voices still insisting that scaling is “all you need” are those with something to sell. The field is maturing, moving past a singular focus toward a more nuanced, multi-faceted approach to building intelligent systems. This collective awakening, though challenging, is a necessary step for meaningful progress.
(Source: Gary Marcus / Substack)
