2019, in 14 pieces

Overturned research practice centered on static-graph frameworks

An SSD-based ANN that broke HNSW's constraint of having to hold everything in memory

Reduced the K/V heads to one, redefining the KV cache as the decoding bottleneck

Provided the yardstick for the later AGI debate

The first large-scale standard corpus built by cleaning Common Crawl with heuristics

Removed the memory barrier to 100B-class training

The standard implementation of large-scale model parallelism

Turned sentence embeddings into a searchable asset

A format that removed FP16's scaling burden. Later the de facto default training precision for the TPU, Ampere and Hopper generations

The first instance of the cycle from benchmark saturation to replacement

The EU's legal basis for governing AI training data

Scripture for the scaling camp. Repeatedly cited as the basis for justifying compute over the injection of human knowledge

Suggested that natural-language instructions can serve as task specifications, and opened the discourse on the risks of releasing models

The prototype of the reranker lineage. Standardized the two-stage structure that recovers, at a later stage, the precision dense embeddings miss