2017, in 12 pieces

Broke kernels down into composable parts. FlashAttention, Transformer Engine and many modern attention kernels are built on top of it

Planted the generational vocabulary of Software 1.0/2.0 and became the prototype for every later Software N.0 discussion

The prototype of mixed-precision training. The whole lineage of shrinking precision, FP16→BF16→FP8→FP4, is a variation on these two techniques

Self-attention alone, without recurrence, made parallel training possible, establishing the architectural premise for large-scale pretraining

Performed translation with convolutions alone, showing the race to move beyond recurrence just before the Transformer

Disclosed the design and measured performance of the first-generation TPU, becoming the reference document for discussions of dedicated accelerators

A design that trains without paired data, lowering the data requirements of generative models

Proposed architecture-agnostic meta-learning, opening up research on few-shot adaptation

The starting point of the ML compiler race. The layer that made TPUs practical and the foundation of all of JAX

The library that brought large-scale vector search into the realm of practical use

Redefined the instability of GAN training as a problem of the distance function

The first demonstration, through conditional computation, of an architecture whose total parameters far exceed its active parameters; the ancestor of every MoE