2021, in 24 pieces

An algorithm registry system and the right to refuse personalized recommendations

Diffusion in latent space rather than pixel space sharply cut compute; the direct basis of Stable Diffusion

The follow-up that brought late interaction down to practically sized indexes

A redesign of FSDP1. The precondition that made FP8 training, partial freezing and compiler optimization possible

The 137B model outperformed the 175B GPT-3 zero-shot on 20 of 25 tasks

Extrapolates with only a linear bias and no position embeddings; one of the two main approaches, alongside RoPE

Opened kernel writing up from a handful of experts to ordinary researchers

The strand that made lexical retrieval learnable by neural networks. Broke down the dense-versus-sparse dichotomy

An event showing that the vector DB could stand as foundation-governed open-source infrastructure rather than a vendor product

Freezes the weights and trains only low-rank matrices, cutting trainable parameters to 1/10,000 for GPT-3 175B, with no increase in inference latency

Presented by Naver as Korea's first hyperscale AI and a means to AI sovereignty; a closed model later succeeded by HyperCLOVA X

Unlike most language models it was trained on dialogue, and Google said fine-tuning after training could significantly improve the sensibleness and specificity of its responses

The starting point of the EU's first comprehensive AI regulation

The start of the path that brought vector search into existing Postgres rather than a separate DB

Rotary position embedding became the foundation of every later context-extension technique

A decisive rebuttal of the narrative that dense retrieval had beaten lexical retrieval. The practical grounds for hybrid search becoming the default

Got around the GPU memory wall by extending into the storage tier. Established the framing that the constraint is bandwidth, not memory

The first case of an AI lab CEO personally designing and putting forward a redistribution policy

"Stochastic parrot" became the standard vocabulary of the LLM-skeptic camp. Set the baseline for the debate over understanding and meaning

The first commercialization of the vector DB as a standalone product category. Triggered the market formation of Weaviate, Qdrant, Chroma and Milvus

The start of the lineage that lowered the hardware barrier to entry for training large models

Simplified MoE and became the standard reference for routing instability and load-balancing loss

The starting point of text-to-image generation

Later the text-conditioning backbone of image generation and retrieval models. The contrastive training on 400 million pairs is based on the paper arXiv:2103.00020