2014, in 9 pieces

The optimizer that later became the de facto standard. Changed the defaults of training recipes

The idea of handling memory through attention-based addressing. The start of the memory-augmented lineage

The first layer that spared frameworks from writing their own kernels

Formalized the encoder-decoder that maps a sequence to a sequence. The ancestor of today's generative model interface

Standardized the design of stacking small 3x3 convolutions deep

The paper that solved the fixed-length vector bottleneck with attention. The direct predecessor of the Transformer's self-attention

Proposed an architecture that uses the competition between a generator and a discriminator as the learning signal, opening up one pillar of generative modeling

Brought the encoder-decoder architecture and the GRU to machine translation, laying the groundwork for attention research

Laid the organizational foundation that would later produce DQN and AlphaGo