Inference Infrastructure & Serving

It is the first model from a new US lab. Company-reported scores were SWEBench Verified 81.6%, GPQA Diamond 90.6%, AIME 2026 97.7% and Terminal Bench v2.1 80.1% (on BF16), and it ships in three versions, BF16, FP8 and NV

Because these are activated LoRAs, the KV cache built by the base model can be reused for the judgment that follows, and IBM said GPU memory with all three components added stays close to that of the base model alone. It

The key is that speed has been split off into a separately billed tier. When latency becomes the bottleneck of agent loops, tokens per second become a product separate from quality, and investment in inference infrastruc

Disaggregated serving has come down from a research claim to an operational guide. The key is using p99 latency rather than average throughput as the yardstick, since agent loops are directly exposed to tail latency. It

A mid-tier update released alongside Anthropic's Sonnet 5.5 in the same week. That two labs simultaneously touted "nearly the same work at 1/5 of the frontier price" shows the competitive axis of the second half of 2026

This rundown takes apart the announced metrics of a frontier lab's new top-tier model item by item, then tests them against four outputs made from the same prompt. The change singled out as most important is not the scor

The back-and-forth over a model that, instead of answering a question in long sentences, returns only a number for how likely each predefined option is to be correct is laid out with the arguments of both sides. The crit

A change that brings to the API surface the fact that the cost of long-running agent loops depends on cache design. Taking tool-list changes and reasoning-effort adjustments off the list of reasons for cache invalidation

With transcription prices down to 10–20 cents an hour, voice enters the range where all of it gets stored, like logs.

A case of pushing the diagnosis that the bottleneck for agent workloads has shifted from compute to KV-cache storage and bandwidth through the entire model design. Using different active parameters for prefill and decode

Full-duplex voice came from a Chinese lab as well, in the same week as OpenAI's GPT-Live-1 (Sep 10) and Google's Gemini 3.8 Live (Sep 15) — a stretch in which the voice interface settles from one lab's differentiator int

Leaves a first-hand record that the bottleneck of AI products has moved from models to stateful infrastructure, and is a case of a coding agent actually being put to work rewriting the company's core infrastructure.

This conversation answers the conventional wisdom about the race to secure accelerators, that whoever has the most wins, by agreeing with the premise but saying that is no longer the case. The example given as evidence i

A trend in which, as NVLink Fusion absorbs even competing inference silicon, NVIDIA's rack architecture is hardening into the de facto standard interconnect for AI data centers.

An inference-architecture shift in which, as long multi-turn and agent workloads become standard, the KV cache extends beyond GPU memory into the storage hierarchy.

A record, in concrete numbers, of AMD GPUs being optimized in a major open inference engine into a real alternative to NVIDIA.

As agentic coding becomes the representative workload of inference serving, the goal of benchmarks itself has shifted from one-shot throughput to the economics of long multi-turn sessions.

A case showing that competition in inference optimization has moved down from schedulers and batching to the level of GPU kernel redesign.

An attempt to handle contexts in the 1 million-token class on a single node; solutions to the memory bottleneck of long-context agents are converging on a combination of sparse attention and hierarchical memory.

Shows a path by which a non-mainstream accelerator joins the mainstream inference stack through a plugin that leaves the vLLM core untouched — the barrier to hardware diversification is coming down.

Overturns the conventional wisdom that "errors in recurrent state accumulate over long contexts", arguing that the recurrent half of a hybrid LLM is in fact the easier half to quantize, and offers four mechanistic explan

Raises a counterargument against the entire lineage of importance-score-based KV eviction: "the scores themselves are unnecessary". The authors explain that only the prompt is fragile, while reasoning sequences sustain t

A case of attacking, under an open license, the point that the bottleneck of on-device inference is kernel implementation rather than models. Per the company's report, it is 2.57 times faster than ORT WebGPU by geometric

A strand that, instead of P/D disaggregation, which requires separate device pools and KV-cache transfers, overlaps P and D on the same devices by isolating only collective-communication order and state while sharing wei

Instead of feeding in a long video whole, the model explores it on its own — multimodal input is folded into the agent loop

The competitive axis for frontier models shifts from coding to autonomous scientific research. Tiered deployment that opens the top model only to screened users has hardened into a product line, and the fact that the pri

A rebuttal paper that sets a baseline — "a sliding window that needs no post-training is enough" — against the strand of linear-attention retrofits that had been in the spotlight as a fix for quadratic scaling. The autho

A low-latency unit for agents is integrated into the rack

Multi-node model loading and weight swapping (RL rollouts) are promoted to first-class problems of the serving stack

72 GPUs per rack, a single scale-up domain and 31TB of HBM4 — alternative silicon entering real-world deployment

Integrates the Vera CPU, Groq 3 LPX, BlueField-4 and Spectrum-6, with agent workloads as an explicit target

The stage at which an on-device runtime covers not just mobile OSes but browsers with the same stack. Figures are the vendor's own measurements

Split training and inference silicon for the first time; a signal that the era of general-purpose accelerators is ending

Ironwood went from announcement to actual supply

The second rewrite of the execution layer since the V1 redesign. Serving competition moved from kernels to schedulers and execution cores

The README recommends moving to vLLM, SGLang, llama.cpp or MLX; the serving engine market consolidated into two leaders plus local

The latest generation of the kernel lineage, formally published at MLSys 2026

Non-GPU inference IP is absorbed. Groq continues as an independent company

A TPU for the inference era. Actual GA came on Mar 31, 2026

Inference on 200B models and fine-tuning of 70B models on a desktop; local machines become agent runtimes

Puts fine-grained sparse attention into a commercial API and cuts long-context costs

A precise standardization of 4-bit precision

The serving engine went from research project to infrastructure public good

An open-source reproduction of frontier-class MoE serving

Quantization-aware training became a standard step in open-weight releases

Explicitly targeted reasoning models, making PD disaggregation, KV routing and autoscaling a vendor standard

The first quantitative measurement breaking down, task by task, the steep drop in token prices at equal performance

The first primary source on the unit economics of inference. DeepSeek itself cautioned that this is an upper bound assuming all usage is paid, and that actual revenue is much lower

Released, one per day over 5 days, working proof of large-scale EP, PD disaggregation and FP8

Made prefix caching the default; up to 1.7× faster than V0

A strategy of lowering switching costs by adopting compatibility with competitors' APIs

The fastest build-out in the industry, by the company's own announcement. But the initial grid connection was only 8 MW, so it relied on temporary gas turbines, and was stepped up to 150 MW only in November

Used wafer-scale SRAM to get around the bandwidth limits of GPU HBM, popularizing the category of inference-only chips

Tiered KV cache management grounded in a production service

A paradigm that treats a whole rack as one giant GPU

Solved the problem of long prompts stalling decode; 2.6–5.6× the serving capacity of vLLM

Splits prefill and decode into separate GPU pools so that latency metrics can be tuned independently

Brought a cache layer for workloads with overlapping prompts, such as multi-turn and agents, down to the runtime level

Automatically reuses the prefix KV cache through a radix tree

The first proposal in the prefill-decode disaggregation line

The first product to put memory capacity and bandwidth, rather than compute, forward as its generational differentiator

Integrated in-flight batching, paged attention and FP8 into the vendor stack

A single-file distribution format with metadata and backward compatibility — the container image of local model distribution

Manages the KV cache like OS paging, cutting memory waste to under 4%; the starting point of the open-source serving standard

Quantization that protects only the top weights as judged by activations; TinyChat runs more than 3× faster than HF FP16

Dependency-free inference in pure C/C++ opened the way to running LLMs on a laptop without a GPU

An independent formulation of the same technique, with a 2–2.5× speedup

Sidesteps the memory-bound nature of decoding without changing the output distribution

The prototype of continuous batching