vLLM
Open-source inference engine for serving language models fast
Records 9
Newest first. Open one to see it with its source.
Disaggregated serving has come down from a research claim to an operational guide. The key is using p99 latency rather than average throughput as the yardstick, since agent loops are directly exposed to tail latency. It
An inference-architecture shift in which, as long multi-turn and agent workloads become standard, the KV cache extends beyond GPU memory into the storage hierarchy.
A record, in concrete numbers, of AMD GPUs being optimized in a major open inference engine into a real alternative to NVIDIA.
As agentic coding becomes the representative workload of inference serving, the goal of benchmarks itself has shifted from one-shot throughput to the economics of long multi-turn sessions.
An attempt to handle contexts in the 1 million-token class on a single node; solutions to the memory bottleneck of long-context agents are converging on a combination of sparse attention and hierarchical memory.
Multi-node model loading and weight swapping (RL rollouts) are promoted to first-class problems of the serving stack
The second rewrite of the execution layer since the V1 redesign. Serving competition moved from kernels to schedulers and execution cores
A sign that open-source serving infrastructure, once text-only, is expanding to cover non-autoregressive and multimodal generation
Made prefix caching the default; up to 1.7× faster than V0