2020, in 13 pieces

The diversity-curation paradigm. At the same time, it passed copyright burdens such as Books3 on to later models

Became the de facto standard for measuring the breadth of knowledge of pretrained models, and an essential metric in model announcements for the next 5 years

Redesigned the quantization loss function around inner-product estimation error, pushing the quality-speed frontier of MIPS forward by a generation

Scaled MoE up to a 600B-scale system and established expert parallelism and automatic sharding

Made the approach of programming by showing examples into an official product interface

Introduced in-context learning, which performs a task from examples in the prompt alone, without updating parameters

The canonical text of retrieval-augmented generation, combining parametric and non-parametric memory

The starting point of the late interaction lineage, distinct from single-vector retrieval

Flipped the default from sparse retrieval (BM25) to dense retrieval

The first study to tackle the cost of embedding migration head-on. Framed as a problem the structural debt of having to re-embed the entire corpus every time the model is replaced

The common foundation shared by XLA, TVM, Triton and Mojo. Ended the practice of writing a new compiler for every accelerator

The layer that made extremely large-scale training possible without rewriting code for model parallelism

Showed that loss can be predicted as a power law of model size, data and compute, becoming the rationale for large-scale investment