Taking vLLM Apart: A Practical Guide to Disaggregated Serving
Published "Taking vLLM Apart: A Practical Guide to Disaggregated Serving" — sets out, with practical figures, prefill/decode disaggregation and moving CPU work off the GPU. On 2 L40S 48GB GPUs with Qwen2.5-7B-Instruct and prompts of about 8k tokens, the p99 inter-token latency (ITL) of a collocated deployment jumps from 23ms to 169ms at 0.4 req/s and reaches 263ms at 2 req/s, whereas prefill/decode disaggregation holds 25–52ms across all load levels. On AMD MI300X with Qwen3-235B, requests meeting both a 1-second TTFT and a 50ms ITL numbered 73/100 disaggregated vs 30/100 collocated. On multi-node H200, mean end-to-end latency fell 59% and P95 latency 67% compared with collocated replicas. Presents three kinds of separable work: (1) prefill/decode disaggregation (transferring about 3GB of KV cache over RDMA for a 10k-token Llama-3.1-70B prompt); (2) a GPU-less render layer that handles tokenization, chat templates and parsing (about 15ms of CPU for a 9k-token prompt); and (3) bidirectional KV transfer that reuses the cache in multi-turn conversations.
Why it matters
Disaggregated serving has come down from a research claim to an operational guide. The key is using p99 latency rather than average throughput as the yardstick, since agent loops are directly exposed to tail latency. It is the open-source side's response to the same pressure that led OpenAI, the same week, to split speed off into a separately billed tier.
Source — Primary sources first; secondary reports are labelled “Press”
Officialvllm.aihttps://vllm.ai/blog/2026-09-29-disaggregated-serving-guideAdded in Week 41, 2026 · See that week