Researchers release SMELT
SMELT released — points out that evaluations of looped transformers, which repeat a block of shared layers to increase effective depth, usually compare at a fixed model size and so mix up architectural advantage with extra FLOPs; presents a recipe (Sparse MoE Transformer, middle layers Loop Twice) that runs the middle half of an MoE transformer's layers twice while matching all three budgets — FLOPs per token, total non-embedding parameters and KV cache. Scaled to four sizes of up to 54B non-embedding parameters and fitted Chinchilla-style scaling laws separately for each architecture
Why it matters
A study that judges, at the level of scaling laws, whether depth reuse (looping) still pays off once budgets are strictly matched. In the authors' own reporting, it saves 6.8–18.0% of training FLOPs at the compute-optimal frontier; the gains are largest on code and grow with sample length and the number of in-context examples; and it offers a mechanistic analysis showing that the second visit reduces attention sinks and reallocates attention mass to content-relevant tokens.
Source — Primary sources first; secondary reports are labelled “Press”
Paperarxiv.orghttps://arxiv.org/abs/2609.01343Added in Week 36, 2026 · See that week