Google updates Android Bench with new LLMs, Gemini still lags
▼ Summary
– Google’s Android Bench, a benchmark for evaluating LLMs in Android app development, has been updated with new models and a more user-friendly framework.
– The benchmark now includes metrics like cost and efficiency, alongside open-weight models, to better assess AI agent performance on 100 development tasks.
– Eight new models were added to the test, including Claude Fable 5, Claude Sonnet 5, and Claude Opus 4.8.
– Google’s Gemini 3.1 Pro ranks fifth on the updated leaderboard, behind GPT 5.4, Claude Sonnet 5, and Claude Fable 5.
– Claude Fable 5 leads the benchmark with 84.5 percent accuracy, significantly outperforming other models.
Code generation has quickly become one of the most common uses for large language models (LLMs), yet no two AI agents handle every development task with equal skill. Earlier this year, Google introduced a benchmark designed to measure how well these models perform in Android app development. Now, Android Bench is receiving a significant update. The leaderboard now includes a broader selection of new models, and the company has rolled out a revamped framework that makes it simpler for developers to engage. Google is encouraging developers to run their own tests and provide feedback that could help refine the benchmark’s future direction.
Even as LLMs gain traction as coding assistants, they are far from infallible. Sorting useful outputs from unreliable results means picking the right tool for the job. Android Bench was built to highlight which AI agents truly excel across a set of 100 distinct Android development tasks. Since its launch in March, Google has expanded the benchmark to include new metrics like cost and efficiency, along with support for open-weight models.
To keep the benchmark current, Google is updating the test suite with eight new models, including some of the most prominent names in the field: Claude Fable 5, Claude Sonnet 5, Claude Opus 4.8, GLM 5.2, Kimi K2.7 Code, MiniMax M3, Qwen 3.7 Plus, and Qwen 3.7 Max.
When Android Bench first launched, Google’s own AI models were already trailing behind OpenAI’s latest offerings. With this expanded lineup, the situation for Gemini has grown even more challenging. On the updated leaderboard, Gemini 3.1 Pro sits in fifth place, behind GPT 5.4, Claude Sonnet 5, and Claude Fable 5. Notably, Fable 5 lives up to the buzz, claiming a commanding lead with an 84.5 percent accuracy score in the benchmark.
(Source: Ars Technica)
