The Download: LLMs’ next leap and AI research’s new direction
▼ Summary
– Transformers are a bottleneck for large language models because their dense attention mechanism becomes increasingly expensive as text length grows and struggles with tracking large amounts of information.
– Four new ideas are proposed to solve the transformer problem, aiming to make LLMs faster, more efficient, and potentially smarter.
– The article is part of MIT Technology Review’s What’s Next series, which previews future trends across industries.
– AI professors are negotiating new realities in academic research, as highlighted by Grace Huckins’ coverage.
– The AI2050 program, funded by Eric and Wendy Schmidt, supports academics working in AI, and its fellows include leading AI researchers facing this uncertain period.
As large language models continue to scale upward, the transformer architecture that powers them is turning into a serious constraint. The dense attention mechanism at its core grows increasingly costly as input text expands, and these models struggle to retain large volumes of information simultaneously. That bottleneck has prompted researchers to explore fresh approaches that could fundamentally reshape how LLMs operate.
Four emerging innovations aim to tackle this transformer problem head on. If successful, they could deliver models that are not only faster and dramatically more resource efficient, but potentially even more intelligent. The ideas range from rethinking how attention is computed to restructuring the underlying architecture itself, each offering a distinct path beyond the current paradigm.
This analysis comes from MIT Technology Review’s What’s Next series, which scans industries, trends, and technologies to offer an early glimpse of what lies ahead.
Meanwhile, the academic world of AI research is undergoing its own transformation. University professors who specialize in artificial intelligence are finding themselves navigating unfamiliar terrain as the field’s center of gravity shifts.
Last week, I attended a gathering in Mountain View, California, where the Schmidt Sciences AI2050 program, funded by Eric and Wendy Schmidt, brought together some of the most distinguished and promising AI researchers in the field. I moderated roundtable discussions and led media training sessions for the group, which supports academics whose work centers on AI.
The roster of fellows reads like a hall of fame for AI research. Even though not everyone made the trip to the Bay Area, I found myself constantly running into scientists I had interviewed before or whose work I deeply respected. For the university-based researchers who form the majority of the AI2050 cohort, this is an especially uncertain moment. The pressures come from multiple directions: industry labs with vast resources, shifting funding priorities, and questions about how academic research can remain relevant in a field moving at breakneck speed.
The full story explores what these researchers are grappling with and what the future may hold for academic AI. It originally appeared in The Algorithm, our weekly newsletter focused on artificial intelligence, which lands in subscriber inboxes every Monday.
(Source: MIT Technology Review)