Fast Weight Attention for Continual Learning
Fast Weight Attention for Continual Learning released — from the view that recurrent fast-weight memories and selective state-space models turn state transitions into online learning rules, argues that under read-after-write autoregressive semantics, the local fast-memory example revealed at time t by the prefix-prediction objective is the prefix-aligned pair (φ(k_{t-1}), v_t), whereas the commonly used same-time association (φ(k_t), v_t), though causal, optimizes a different internal objective. Derives normalized first-order update rules for squared-error regression and negative inner-product objectives, defining Falcon-1 (scalar NLMS), Falcon-2 (column-wise extension) and Falcon-3 (sliding-window minibatch) and the inner-product variants Falcon-1A/2A/3A, and presents recurrent, masked-parallel and chunk-parallel forms along with a numerically stable positive-decay renormalization
Why it matters
A framework that reformulates the state-update rules of the DeltaNet and Mamba families as a time-alignment problem — "what to associate, and when" — making time alignment, plasticity, forgetting and bounded rehearsal into separable design axes. In the authors' own reporting, the representative variants stay competitive in language modeling and improve length extrapolation on variable-digit addition.
Source — Primary sources first; secondary reports are labelled “Press”
Paperarxiv.orghttps://arxiv.org/abs/2608.27763Added in Week 36, 2026 · See that week