Model Architecture & Training Methods
73 records · 1 notes · records tagged with this as a related axis are included.
Attempts to change what's inside the model
Even after the Transformer became the standard, its insides and surroundings kept changing. Mixture-of-experts designs that switch on only the parts needed, attempts to replace or supplement attention, position encodings for handling long inputs, and techniques that keep training stable all belong on this axis. Most of it never shows up in a product name, but when a model suddenly gets cheaper or can take much longer inputs one day, the reason is usually here.
It starts from the point that looking only at the endpoints lets preference changes from early and late training cancel each other out and hide the parts where the teacher changed consistently. The authors reported that
Company-reported, the 209M model ranks No. 1 in the pretrained zero-shot categories of GIFT-Eval, TIME and fev-bench, and the 28M model beat Chronos-2, which is about four times larger. On an 8-thread server CPU the 5M m
The authors reported that, trained on 12.3 million question-answer pairs on top of Qwen3.5-4B, it ranked No. 1 among the 8 models compared, with an overall 65.64%, and was No. 1 on 7 of 12 benchmarks. They reported it wa
The point at which generation paths outside autoregression matured from the claim that "parallel generation is fast" to the question of "how to preserve dependency structure". A design that tries to solve the respective
An attempt to measure, separately from video generation quality, "whether a model has basic common sense about the physical world". Moving cognitive-development tasks into a benchmark shows world-model evaluation startin
It recasts why agents break down on long tasks — not "because they cannot read the environment" but "because stale state descriptions they wrote themselves linger". It means what a harness has to manage is not tool calls
A case of pushing the diagnosis that the bottleneck for agent workloads has shifted from compute to KV-cache storage and bandwidth through the entire model design. Using different active parameters for prefill and decode
A case of a JEPA-family predictive architecture moving beyond video to molecules, clinical data and weather within the same framework. It reflects a trend toward defining world models not as "video generation" but as "do
A case of solving, through architecture, a chronic limitation of diffusion language models: "state breaks at chunk boundaries". A substantive advance that arrived just as the observation topic "diffusion language models"
A case of the in-context learning paradigm of LLMs being transplanted to robot manipulation, showing robot foundation models heading toward generalization without per-task fine-tuning.
Shows that a fully end-to-end unified architecture with neither a separate vision encoder nor a VAE has reached practical performance, suggesting that the standard design of unified multimodal models may change.
A case that extends LLMs' basic training objective, token-level autoregression, to concept-level latent prediction, offering a path to better pretraining efficiency at the architecture level.
A design that separates generative rendering from state management, pointing the way for world models to move beyond "plausible video" to playable simulators in which rules are enforced.
A structure that feeds the routing logs generated during serving back in as the next training data — an early prototype of a self-improvement loop in which the boundary between deployment and training disappears.
A case showing that competition in inference optimization has moved down from schedulers and batching to the level of GPU kernel redesign.
A turning point at which world-model-based robot policies move from competition among closed systems to a controlled science that checks "which design elements work, and why".
Overturns the conventional wisdom that "errors in recurrent state accumulate over long contexts", arguing that the recurrent half of a hybrid LLM is in fact the easier half to quantize, and offers four mechanistic explan
A strand that, instead of sending the prompt to a remote model every time, distills the specification once and sets it into a local function. In the authors' own reporting, 83.6% semantic accuracy on FuzzyBench-Hard. Pre
Raises a counterargument against the entire lineage of importance-score-based KV eviction: "the scores themselves are unnecessary". The authors explain that only the prompt is fragile, while reasoning sequences sustain t
A strand that puts 3D world state into multimodal models as a native representation rather than an add-on condition. A recent case for the observation topic "world models"; since code, models and datasets were released t
A case of the diffusion language model family crossing over into image generation, adding to the accumulated evidence for the observation topic "diffusion language models". In the authors' own reporting, it claims 53.53
A study that judges, at the level of scaling laws, whether depth reuse (looping) still pays off once budgets are strictly matched. In the authors' own reporting, it saves 6.8–18.0% of training FLOPs at the compute-optima
A case of a frontier open-weight model disclosing the rationale for its architecture choices ablation by ablation. In the authors' own reporting, loss and downstream accuracy do not always move together, and combining th
The material starts from the contrast that the human brain runs on about 20 watts while an AI supercomputer uses 20 million watts, and traces the root of that gap to the separation of processor and memory in the 1945-sty
A rebuttal paper that sets a baseline — "a sliding window that needs no post-training is enough" — against the strand of linear-attention retrofits that had been in the spotlight as a fix for quadratic scaling. The autho
A framework that reformulates the state-update rules of the DeltaNet and Mamba families as a time-alignment problem — "what to associate, and when" — making time alignment, plasticity, forgetting and bounded rehearsal in
A strand that directly tackles the problem of self-improvement loops wearing down performance by reusing past successes uncritically. In the authors' own reporting, with a 4B model across three domains — financial reason
Opens a parametric knowledge-graph strand that, unlike the GraphRAG lineage, which queries the graph at retrieval time, compiles the graph into model weights in advance — while at the same time pinning down that this str
Breaks the assumption that N and D are independent, reducing prediction error by 1.5 to 3 times and extrapolating with about 10 times less compute
Shows a path to obtaining a diffusion model with less than 10% of the original token budget; about 1,500 tok/s on a single H100
An LLM lab expanding into replacing industrial simulation. No named model release at the time of the announcement
At 27% of the per-token FLOPs and 10% of the KV cache of V3.2, it made 1M context something supported all the time
Brought the trend of merging separate model families into a single set of weights to open weights
Recasts the optimizer as an associative memory module and proposes a sequence model that learns a self-update algorithm
The route of working around long-context limits through architecture rather than retrieval
A follow-up document to the July release. A case of an on-device family having public documentation at the level of a technical report
The first to beat full attention in a fair comparison; up to 75% less KV cache and up to 6× faster decoding at 1M context
An early case of merging reasoning and non-reasoning models into a single set of weights — a direction that later became the industry standard
Claims up to 10× the speed of speed-optimized frontier models. The first time a diffusion LM entered a commercial track
A non-reasoning model that does not think before responding; company-reported GPQA 71.4% and SWE-bench Verified 38.0% trailed o3-mini (high) at 79.7% and 61.0%, while it led on MMMLU (85.1% vs 81.1%) and SWE-Lancer Diamo
A turning point toward efficiency for a Korean lab
Moved sparsity from the inference stage to the pretraining stage
Reopened the question of whether autoregression is a necessary condition for LLM capabilities at all
Established low-precision pretraining at frontier scale
Processes bytes directly, without a fixed vocabulary, using entropy-based dynamic patches; the first large-scale scaling of the tokenizer-free approach
Set new training-speed records over AdamW using Newton-Schulz orthogonalization
Showed that the conflict between the two laws stems from counting non-embedding parameters and from analysis at small scale
A third path, distinct from MQA and GQA, that compresses KV into a low-dimensional latent
Showed a path to non-English capability by combining existing open weights without pretraining — the prototype strategy for countries at a compute disadvantage
Included inference cost in the objective, providing the theoretical basis for today's overtraining norm
A model from a Korean startup that, per Upstage, topped the Hugging Face Open LLM Leaderboard at release; the 74.2 average belonged to the instruction-tuned SOLAR-10.7B-Instruct, while the base model scored 66.04
With linear-time sequence modeling, made what comes after the Transformer a serious option
The finding that initial tokens act as attention sinks; 2 years later gpt-oss adopted learnable sinks
The de facto standard recipe for context extension, unifying Position Interpolation and NTK-aware interpolation
Restructured the RNN so it can be trained at Transformer level; the opening signal of a revival of linear complexity
A compromise between MHA and MQA that allows uptraining of existing checkpoints, and became the industry-standard KV-saving technique
A rebuttal arguing that emergence is an artifact of discontinuous metrics; with continuous metrics the curves are smooth
Popularized the idea that a mismatch between the tokenizer and the model's training corpus is a real failure mode
Adding and subtracting task vectors to inject or remove capabilities made model merging something you can compute
Its claim that abilities emerge at scale thresholds became a core narrative of the scaling discourse
The starting point of the diffusion language model lineage
The first large-scale use of the Pathways system; reported surpassing the few-shot performance of prior large models such as GPT-3, Gopher and Chinchilla on 28 of 29 English NLP tasks
Exposed that the large models of the time were undertrained; the 70B model outperformed the 280B Gopher
Demonstrated that simply averaging the weights of several fine-tuned models raises performance; the starting point of model merging
Transferred hyperparameters tuned on small models zero-shot to large ones, making the practice of getting it right in a single run possible
Extrapolates with only a linear bias and no position embeddings; one of the two main approaches, alongside RoPE
Rotary position embedding became the foundation of every later context-extension technique
Simplified MoE and became the standard reference for routing instability and load-balancing loss
Scaled MoE up to a 600B-scale system and established expert parallelism and automatic sharding
Reduced the K/V heads to one, redefining the KV cache as the decoding bottleneck
Fine-tuned with just one additional output layer, it set new state-of-the-art results on 11 NLP tasks, including GLUE 80.4% and SQuAD v1.1 F1 93.2 (as reported by the authors)
Reported that combining unsupervised generative pre-training with supervised fine-tuning improved the state of the art on 9 of the 12 tasks studied
The first demonstration, through conditional computation, of an architecture whose total parameters far exceed its active parameters; the ancestor of every MoE