Training Data & Post-Training
64 records · 2 notes · records tagged with this as a related axis are included.
What was it fed, and what was it polished with afterward?
A model's character comes more from its data than from its structure. This axis gathers where training data came from and how big it was, the use of synthetic data, and the stages after pretraining — post-training that polishes a model with human feedback, trains preferences in directly, or adds specific skills on top. Fights over copyright and sources often straddle this axis and the regulation axis at once, so related events are linked to each other.
Company-reported, it covers 34 language and script combinations with 2.601 trillion tokens in native scripts plus 224.6 billion in Latin script, and includes 438.4 billion restored tokens kept only where the quality scor
It starts from the point that looking only at the endpoints lets preference changes from early and late training cancel each other out and hide the parts where the teacher changed consistently. The authors reported that
The point at which generation paths outside autoregression matured from the claim that "parallel generation is fast" to the question of "how to preserve dependency structure". A design that tries to solve the respective
In a company presentation on sovereign transformation in healthcare, sovereignty is broken into three layers, data, model and operations, and its essence is defined as the freedom of choice to swap out whatever you curre
The first publicly disclosed large-scale extraction attempt since hiding reasoning traces became a trade-secret protection measure. Because the attribution points to a specific Chinese lab, it reads as a geopolitical eve
A case of building a public good to prepare for the next outbreak by loading AI-generated predicted structures wholesale into a public database. The starting point of pandemic response shifts from "solving structures thr
A case of AI moving from summarizing the literature to "deciding for itself what to look at and noticing anomalous patterns first". A structure in which a lab sets up its own wet lab to check its own model's discoveries
Close to the first case of a lab entrusting an outside committee with how to release its own results to academia. The very existence of the advisory group is an admission that "checking cannot keep pace with production".
The key is that the target of self-improvement is the harness configuration, not the model weights — a measurement showing there is still room to cut costs by a third with the same model.
A design that seeks to draw self-improvement out of ongoing collaboration with researchers rather than out of benchmark scores. Together with ModularRSI and SoL-Pi from the same week, it forms a trend in which "the targe
A head-on attack, through data separation, on the chronic problem of harness self-improvement: "fitting only the benchmark". Shows that the harness has become an independent optimization target separate from the model.
At an event unveiling an autonomous-driving development and operations system, the key idea singled out is a framing: the speed of the data flywheel is determined not by the sum but by the product of scale, quality, effi
An attempt to break RSI down into stages and make it something to measure rather than a slogan. The theoretical axis of this week's simultaneous cluster of RSI and harness self-improvement papers on arXiv.
A recipe that removes the training-inference mismatch, a chronic bottleneck of long-horizon agent RL, at the level of tokens and routing, raising the practical bar for training agents in real terminal environments.
A case that extends LLMs' basic training objective, token-level autoregression, to concept-level latent prediction, offering a path to better pretraining efficiency at the architecture level.
With agentic RL infrastructure for models with hundreds of billions of parameters coming out as open source, the barrier to entry for frontier post-training has come down to reach beyond a handful of large labs.
A structure that feeds the routing logs generated during serving back in as the next training data — an early prototype of a self-improvement loop in which the boundary between deployment and training disappears.
A companion paper to Terminal-Universe above, released the same week, splitting the problem of making environments (synthesis) from that of growing them (evolution). In the authors' own reporting, gains on Terminal-Bench
An in-house test of the automated-researcher route, which delegates alignment research itself to models. An attempt to shift the bottleneck of safety work from human researchers' time to compute, while monitoring researc
A strand that directly tackles the problem of self-improvement loops wearing down performance by reusing past successes uncritically. In the authors' own reporting, with a 4B model across three domains — financial reason
A case of opening an external checking channel into a structure in which vendors monopolize data on actual usage. The design spells out the freedom-to-publish clause and the third-party audit in order to set it apart fro
The court found the output-infringement allegations sufficient; the DMCA claims survive
About 400,000 books at $3,000 per book; more than 91% of eligible authors filed claims
The dispute widens from fair use to the preservation and concealment of evidence
The human authorship requirement for works generated by AI alone is settled
The distillation dispute moves from suspicion to quantified accusation. Choosing to disclose its detection work and seek an industry-wide response rather than sue reflects the limits of legal remedies
Explicitly applied the commercial TDM exception and found that a natural-language opt-out did not meet the machine-readability requirement. Not final, as an appeal to the BGH was permitted
The first controlled comparison showing distributed training is not just possible but equal or better
Released the data and the post-training pipeline together
The first case in Europe to treat memorization as reproduction and to rule out the TDM exception
Seven cases, including those brought by the NYT and the Authors Guild, combined into one MDL
Matches continual pretraining with up to 100 times fewer tokens — a direct technical answer to data exhaustion
Distributed training moves beyond pretraining demos into the post-training path of real product models
The bottleneck shifts from static datasets to interactive environments that check outcomes. At the same time, skeptics point to reward hacking and rapid obsolescence
Unlike the usual practice of opening only the weights, it released the post-training data too. A baseline for reproducibility in the open community
The data-side response to litigation risk; reached Llama 1 and 2 7B-class performance with openly licensed data alone
The largest pretraining run over the internet. Solved collaboration among untrusted participants with blockchain coordination
Declined to set out definitive rules and introduced a market dilution theory. As of August 2026 the final version has not been published
ToS-based claims of output ownership were raised to a national policy issue for the first time
Proved that reasoning ability is not a secret recipe of closed labs; self-checking, re-examination and strategy switching emerged. The same report also published the RLVR recipe and a recipe for transferring the capabili
Claims at the level of a preliminary report turned into a reproducible paper and code
The turning point from preference learning to verifiable rewards
The starting point of the approach of training over internet bandwidth, without data center interconnects
Estimated 400 trillion to 20 quadrillion effective tokens by 2030, and laid out the order of constraints: power binds first, then chip production capacity
The peak of the model-collapse discourse
A subset scored for educational value by an LLM classifier produced better models than the original, standardizing model-based filtering
The data-generation pipeline itself became the product
Distillation became so cheap that it no longer even needs API calls
The quantitative reference point for the data-exhaustion discourse
The last simplification in the direct alignment family
Raised the upper limit on the scale of open web corpora
Proved that replacing real data leads to collapse but accumulating it keeps a finite upper bound on error, redefining the model-collapse discourse
Representative of the line of work that simplifies RL
Group relative policy optimization without a value function later became the de facto standard for RL in reasoning models
Concluded that no loss function is universally superior and that the inductive bias that suits the setting is best
Released the data together with the tools, making data science reproducible
Rephrasing or translation alone defeats n-gram decontamination, and 8–18% of HumanEval overlaps with public corpora
A proposal to make measuring contamination for each benchmark a community standard
Trained a 1.3B model on 7B tokens to reach 50.6% on HumanEval, demonstrating that data quality matters more than quantity
Claimed that well-filtered web data alone can outperform curated corpora; MinHash fuzzy deduplication became a standard technique
An open reproduction of the LLaMA training recipe that opened an era of reverse-engineering the data behind closed models
A bootstrap in which a model creates its own training data; the starting point of the synthetic data industry
Unlike most language models it was trained on dialogue, and Google said fine-tuning after training could significantly improve the sensibleness and specificity of its responses
The diversity-curation paradigm. At the same time, it passed copyright burdens such as Books3 on to later models
The first large-scale standard corpus built by cleaning Common Crawl with heuristics