Sliding-window beats linear attention
Sliding-window beats linear attention released — points out that the line of research retrofitting existing LLMs with linear attention has never been properly compared with simpler baselines, and reports that sliding-window attention (SWA) with sinks matches or beats post-trained linear-attention models across several LLMs and a range of downstream tasks. On long-context reasoning tasks such as Needle-in-a-Haystack and BABILong, SWA performed 2–10 times better than linear attention; it claims that SWA needs no post-training, is very fast and uses less memory
Why it matters
A rebuttal paper that sets a baseline — "a sliding window that needs no post-training is enough" — against the strand of linear-attention retrofits that had been in the spotlight as a fix for quadratic scaling. The authors conclude that linear-attention models will catch up with SWA only if trained from scratch or put through extensive post-training. Figures are as reported by the authors.
Source — Primary sources first; secondary reports are labelled “Press”
Paperarxiv.orghttps://arxiv.org/abs/2608.28444Added in Week 36, 2026 · See that week