2016, in 14 pieces

The paper that turned architecture design itself into a search problem

An alternative that processed sequences with dilated convolutions instead of recurrence. One branch of the race to move beyond recurrence

The point at which neural machine translation crossed over from research to commercial service

A system description of attention-based seq2seq implemented at production scale

Showed that autoregressive generation extends to audio. The foundation for later TTS and audio generation

With imperative execution, began to change the framework landscape in the research community

Normalization that does not depend on the batch. Once the Transformer adopted it, it became a de facto basic component

The standard technique for training on sensitive data, and a technical reference point for later regulatory discussions

The start of the path toward silicon dedicated to neural networks rather than general-purpose GPUs

The standard layer that optimizes all-reduce for the hardware topology. All data-parallel, ZeRO and FSDP training runs on top of it

The default indexing algorithm of almost every vector DB today

An event that changed public perception of what AI can do. It became the backdrop for later discussions of investment and policy

Opened up the path to on-device training on mobile

Revealed the combination of policy and value networks with Monte Carlo tree search. Part of the lineage that combines search and learning