On the Design of Qwen3.8-Next Architecture
On the Design of Qwen3.8-Next Architecture released — describes the design of Qwen3.8-Flash-Next, a sparse MoE model with 125B parameters and 6B active per token, plus a 51B n-gram embedding table kept off the accelerator. It combines a layer-wise hybrid of Gated DeltaNet (GDN) and global attention, with full attention in only one of every four layers, with Qwen Sparse Attention and Gated Residual, and evaluates each design change along three axes: its effect on loss and downstream benchmarks, on training, prefill and decode costs, and on optimal hyperparameters
Why it matters
A case of a frontier open-weight model disclosing the rationale for its architecture choices ablation by ablation. In the authors' own reporting, loss and downstream accuracy do not always move together, and combining this architecture with the Muon optimizer pushes the optimal learning rate and batch size upward, making batch-size warmup unnecessary and greatly improving stability in stress tests.
Source — Primary sources first; secondary reports are labelled “Press”
Paperarxiv.orghttps://arxiv.org/abs/2608.30320Added in Week 36, 2026 · See that week