Kog pushes GPUs deeper to boost inference efficiency
▼ Summary
– French startup Kog claims standard datacenter GPUs can achieve extremely fast inference through software optimization, demonstrating 3,000 tokens per second with its open-sourced 2-billion-parameter model Laneformer 2B.
– Kog targets enterprises frustrated by slow AI workflows, like Claude Code users, and expects software engineering to be its first major use case, though it pivoted to focus on larger models after customers showed reluctance to fine-tune small ones.
– CEO Gaël Delalleau plans to prove the approach works on large language models by September, aiming for 10x speed to secure a Series A round.
– Delalleau’s background in solid-state physics and offensive cybersecurity drives Kog’s hands-on method of reverse-engineering GPU hardware at a low level, though this limits the team to a few chips at a time.
– Kog faces competition from ZML’s hardware-agnostic software but differentiates with deeper GPU acceleration research; it also benefits from European support via Scaleway, Bpifrance, and French Tech 2030.
The race to accelerate AI inference is heating up, and while Cerebras earned a warm reception from public markets with its custom silicon, a French startup called Kog is making a different bet: that traditional GPUs still hold untapped potential waiting to be unlocked through software.
Kog captured widespread attention in May when its technical preview landed on the front page of Hacker News. The demonstration aimed to show that “extremely fast single-request decoding is possible on the standard datacenter GPUs enterprises already own,” using the AMD MI300X and NVIDIA H200 as proof points. While some observers were disappointed that the technique didn’t apply to consumer laptop GPUs, others recognized the broader implications. As inference speed and cost become critical bottlenecks for AI adoption, Kog’s promise to extract new performance from existing hardware through software optimization generated serious commercial interest. “We had 200 tangible business leads,” CEO Gaël Delalleau told TechCrunch.
Based on early market feedback, the solo founder anticipates software engineering will be the first major use case. Experienced users of Claude Code know the frustration of waiting hours for results, and Anthropic itself has demonstrated that speed commands a premium, charging a price multiple for Claude’s Fast Mode. Kog aims to attract customers who find those delays unacceptable, particularly professionals who depend on AI workflows. The startup also works with design partners who generate games and apps from simple prompts, where faster output through the Kog Inference Engine (KIE) directly translates to increased revenue, Delalleau explained.
The company acknowledges that the market isn’t fully mature yet. Through observing demand, Kog discovered that prospective customers aren’t willing to fine-tune small models. “And that’s why since the launch, we’ve been fully focused on accelerating the development of larger models to meet the demand we’ve seen,” Delalleau said.
That focus represents a significant leap from what the demo actually showed. Kog’s impressive 3,000 per-request tokens per second (TPS) was achieved using a purpose-built small model with roughly 2 billion parameters, the now open-sourced Laneformer 2B. The company’s larger promise of “30x faster LLM inference” remains unproven at scale.
Delalleau dismisses skeptics who doubt the approach can work with large language models, whose size often challenges inference chips. “GPUs have a bright future,” he insisted. He contends that the notion of GPUs being poorly suited for decoding has become a misconception, noting that newer GPUs offer increasing memory bandwidth that simply awaits proper utilization.
Kog isn’t alone in exploring software-driven GPU optimization. ZML, another French company, released hardware-agnostic software that bypasses Nvidia’s CUDA to enable fast inference across competing chips. But Delalleau sees Kog as more aligned with Stanford’s Hazy Research lab, operating at an even deeper level of GPU acceleration.
Delalleau’s path to this work is unconventional. He studied solid-state physics at École Polytechnique before moving into offensive cybersecurity, also known as white hat hacking. This background shaped his approach. On the science side, “there’s this mindset of understanding the laws of physics, and the laws of the GPU in order to make the most of them.” From hacking, the four-time DEF CON CTF finalist learned “to reverse-engineer things at a very low level, down to assembly language and binary code, to understand how it works, and to try to use it to achieve a goal for which it wasn’t necessarily designed.”
This hands-on methodology is intensive and time-consuming. “For every new GPU, we’ll dedicate several weeks or even months, to really dig into the details and conduct GPU engineering research on that hardware,” Delalleau said. With a team of just 11 people, Kog faces practical limits on how many chips it can support in the near term.
Longer term, Kog hopes to codify its approach into agent-based pipelines that could support more chips and models automatically. As Europe pushes to build indigenous AI capabilities, this could provide sovereignty tailwinds for the startup, which already counts Scaleway as a supporter and has backing from France’s Bpifrance and the French Tech 2030 program.
For now, Kog must prove its methodology works on large language models, which will also be crucial for securing additional funding. “Once we’ve implemented our first major model at 10x speed, which I think will be in September, we’ll be able to start demonstrating customer traction and from there, raise our Series A,” Delalleau said.
(Source: TechCrunch)