2022, in 54 pieces

A bootstrap in which a model creates its own training data; the starting point of the synthetic data industry

Turned embeddings into a commodity and removed the cost barrier to the RAG boom

Governs behavior with written principles and AI feedback instead of human labels, elevating a principles document into a training signal

Adding and subtracting task vectors to inject or remove capabilities made model merging something you can compute

A rebuttal to the challenge BEIR had raised. The starting point of the open-weight embedding race

The day the Council's position for the EU trilogue negotiations was locked in — later changes to the text were negotiated against this document

Sidesteps the memory-bound nature of decoding without changing the output distribution

The conversational UI opened prompting to non-experts, turning a skill of a few practitioners into a mass phenomenon

Real-name authentication and labelling duties for deepfakes. The forerunner of generative AI regulation

Arriving the month after LangChain, it shaped another standard vocabulary for the RAG indexing layer

Together with PAL, formed the code-as-reasoning lineage

The lineage of moving computation outside the language model

The work that made comparability between models possible in the first place

Automatically generated instructions matched or beat human-written ones on 19 of 24 tasks

Standardized quantization as a post-processing step

Spread the standard RAG vocabulary of loader, chunking, embedding, vector store and retriever

The common yardstick for the embedding model race. Its key conclusion: no embedding method dominates every task

Redefined compute as a strategic good; the prototype of every later regulation

An early attempt to automate the manual work of writing CoT exemplars

The standard form of the agent loop that combines reasoning techniques with tool use. Cross-referenced with axis 3

Demonstrated that neurons store more features than concepts in superposition; the theoretical starting point of the entire SAE family

Commoditized multilingual ASR, making speech a standard input to LLM pipelines

Foundational vocabulary of the LLM security field. Every later security framework, including the OWASP LLM Top 10, uses the term

As a joint proposal by the three companies, it made FP8 the de facto standard format. A specification agreement that prevented format fragmentation

Made a high-quality generative model that runs on consumer GPUs open-weight, sparking both the LoRA and ControlNet ecosystem and the copyright debate

Made it possible to run large models on consumer-grade GPU servers

Its Discord-based UX brought in non-developer creators on a large scale

The prototype of continuous batching

The moment a training-infrastructure team moved down into inference. The standard for serving large models in the generation before vLLM

The founding document of the JEPA and world-model approach. LLM skepticism took the form of an alternative design rather than criticism

The baseline of the pre-agent era — the limits of autocomplete that suggests code but cannot run or check it on its own

Its claim that abilities emerge at scale thresholds became a core narrative of the scaling discourse

A community-crowdsourced benchmark. Concluded that performance improves with scale but stays low in absolute terms

Effectively opened up the road to long context

The starting point of the diffusion language model lineage

A technique that made embedding dimensions adjustable after the fact

An early case of treating generated reasoning as something to check and to resolve conflicts in

CoT without examples. The original of every later zero-shot reasoning prompt

The prototype of the decomposition-and-planning lineage (Plan-and-Solve, Self-Discover)

A lineage of modular, decomposed reasoning rather than single CoT generation

The prototype of the few-shot VLM, connecting frozen vision and language models through cross-attention

The first large-scale use of the Pathways system; reported surpassing the few-shot performance of prior large models such as GPT-3, Gopher and Chinchilla on 28 of 29 English NLP tasks

Exposed that the large models of the time were undertrained; the 70B model outperformed the 280B Gopher

The orchestration layer that made PaLM-class training possible, and the execution foundation of the JAX/TPU stack

The turning point that made FP8 training a hardware-supported feature. The first case of software and silicon bound together as one

The start of the test-time sampling-and-voting lineage. GSM8K +10%, MultiArith +24%

"Hitting a wall" became the stock phrase of scaling skepticism

Demonstrated that simply averaging the weights of several fine-tuned models raises performance; the starting point of model merging

Transferred hyperparameters tuned on small models zero-shot to large ones, making the practice of getting it right in a single run possible

Showed that outputs of the 1.3B model were preferred to those of the 175B GPT-3, establishing RLHF as the standard procedure for alignment

The point at which existing search engines began absorbing vector search. The starting point of the consolidation debate over whether a dedicated vector DB is needed

Automated the search for parallelization strategies. Redefined manual 3D parallelism tuning as a compilation problem

The starting point of the entire lineage of changing reasoning performance through prompts alone. With 8 CoT exemplars, the 540B model reached SOTA on GSM8K

OpenAI itself wrote that it unlocked capabilities that had been hard to draw out through prompt engineering alone