Inference
Latest news, analysis, and insights about Inference.
Microsoft Maia 300: 40% More Tokens per Megawatt, Ships 2027
Microsoft says Azure Maia 300 will deliver 40% more tokens per megawatt than leading GPU-based systems and up to 7x the rack-scale throughput of Maia 200. Production starts in 2027 as Microsoft scales Maia to multi-gigawatt deployments.
Iterate.ai Lifeboat GA: 2–6× Agent Sessions per GPU
Iterate.ai launched Lifeboat GA, a confidential-compute LLM inference engine claiming 2–6× more concurrent agent sessions per GPU, with company benchmarks of 2,048 sessions on one RTX PRO 6000 Blackwell and pricing from free to $499.99/mo.
Volantis Raises $88M Series A for Photonic AI Inference
Volantis raised an $88 million Series A co-led by Lachy Groom and Abstract Ventures for photonic inference hardware targeting models over 20 trillion parameters at up to 10,000 tokens per second per user, with first systems aimed at 2027.
vLLM's Wide Expert Parallelism Makes DeepSeek Inference 10x More Efficient at Scale
The vLLM team just published benchmarks showing 2,200 tokens per second per H200 GPU for DeepSeek inference. Their 'wide-ep' approach could reshape the economics of serving massive mixture-of-experts models in production.