vLLM releases a Tiered KV Cache Offloading framework
Released a Tiered KV Cache Offloading framework. In a host-centric design in which all KV passes through host DRAM, evicted KV is preserved, instead of being recomputed, in three kinds of auxiliary tier: the file system (shared automatically through content-addressed naming), S3-compatible object storage, and peer-to-peer using ZMQ coordination and RDMA transfers. The company claims that, measured with Qwen3.6-35B-A3B on 2 H100s, throughput was more than 2 times that of the alternatives beyond 128 concurrent conversations; available from vLLM v0.22.
Why it matters
An inference-architecture shift in which, as long multi-turn and agent workloads become standard, the KV cache extends beyond GPU memory into the storage hierarchy.
Source — Primary sources first; secondary reports are labelled “Press”
Officialvllm.aihttps://vllm.ai/blog/2026-09-10-tiered-kv-offloadingAdded in Week 38, 2026 · See that week