Prehistory — From Neural Networks to the Transformer

Performed translation with convolutions alone, showing the race to move beyond recurrence just before the Transformer

Disclosed the design and measured performance of the first-generation TPU, becoming the reference document for discussions of dedicated accelerators

A design that trains without paired data, lowering the data requirements of generative models

Proposed architecture-agnostic meta-learning, opening up research on few-shot adaptation

Redefined the instability of GAN training as a problem of the distance function

The paper that turned architecture design itself into a search problem

An alternative that processed sequences with dilated convolutions instead of recurrence. One branch of the race to move beyond recurrence

The point at which neural machine translation crossed over from research to commercial service

A system description of attention-based seq2seq implemented at production scale

Showed that autoregressive generation extends to audio. The foundation for later TTS and audio generation

With imperative execution, began to change the framework landscape in the research community

Normalization that does not depend on the batch. Once the Transformer adopted it, it became a de facto basic component

The start of the path toward silicon dedicated to neural networks rather than general-purpose GPUs

An event that changed public perception of what AI can do. It became the backdrop for later discussions of investment and policy

Revealed the combination of policy and value networks with Monte Carlo tree search. Part of the lineage that combines search and learning

The launch of the organization that would become the starting point for the GPT series and the shift to commercialization

Made training of very deep networks possible through residual connections. A basic component of most later architectures, including the Transformer

Opened up an internal system, kicking off the framework competition and making deep learning tools an industry standard

Proposed for medical imaging, but later became the basic backbone of generative work such as diffusion models

Formalized distillation, which transfers a large model's knowledge to a small model. The root of today's small-model lineage

The point at which deep reinforcement learning passed peer review and was recognized as an official achievement

Stabilized and sped up the training of deep networks and was adopted as standard in most architectures

The optimizer that later became the de facto standard. Changed the defaults of training recipes

The idea of handling memory through attention-based addressing. The start of the memory-augmented lineage

Formalized the encoder-decoder that maps a sequence to a sequence. The ancestor of today's generative model interface

Standardized the design of stacking small 3x3 convolutions deep

The paper that solved the fixed-length vector bottleneck with attention. The direct predecessor of the Transformer's self-attention

Proposed an architecture that uses the competition between a generator and a discriminator as the learning signal, opening up one pillar of generative modeling

Brought the encoder-decoder architecture and the GRU to machine translation, laying the groundwork for attention research

Laid the organizational foundation that would later produce DQN and AlphaGo

Made it possible to train probabilistic generative models with backpropagation, becoming one pillar of the generative model lineage

An early case showing that deep reinforcement learning works from screen input alone

Established in a paper the combination of GPUs, large-scale data and deep CNNs. The conventional starting point of the current deep learning era

The turning point at which, with a wide margin over second place, deep CNNs became the mainstream methodology of computer vision

Curbed overfitting with simple regularization and later became a standard technique

Framed representation learning as a single problem, becoming the field's standard reference from then on

Separate industry groups confirmed the same shift (HMM→DNN), and deep learning became the industry standard for speech recognition

Showed that high-level concepts emerge from nothing more than large-scale unlabeled data and large-scale compute. A symbol of Google's investment in deep learning

An officially recognized benchmark record of a neural network surpassing humans on a specific visual recognition task

The beginning of a setup in which frameworks determine the speed of research. Influenced the design of later deep learning libraries

An early paper demonstrating the advantages of ReLU, which later became the standard activation function

A standard annual competition in large-scale image classification began. It later became the stage for deep learning's breakthrough

Showed that activation functions and initialization determine training stability. The origin of Glorot initialization

Showed that GPU acceleration alone, without unsupervised pretraining, delivered top performance, reviving the purely supervised learning approach

An early case of applying GPUs to deep learning training in earnest. The starting point of the premise that compute is a resource

Released a large-scale image dataset based on the WordNet hierarchy, creating the recognition that data is the bottleneck

Showed experimentally that layer-wise unsupervised pretraining eases the difficulty of training

Showed once again that deep networks can be trained, sparking the revival of deep learning in the 2000s

Proposed a neural language model that learns words as dense vectors. The direct predecessor of the embedding lineage that includes word2vec

Established a practical CNN architecture combining convolution, pooling and fully connected layers, and made MNIST the standard testbed

Used gates to mitigate vanishing gradients, making it possible to learn long sequences. The standard sequence model before the Transformer

Confirmed in theory and experiments that recurrent neural networks struggle to learn long-term dependencies, motivating the development of later architectures

The first to show that the learning signal vanishes as depth and sequence length grow. The problem definition behind LSTM and residual connections

An early case of combining weight sharing with backpropagation to apply a neural network to a real-world task

Together with Cybenko, established the foundation for the approximation capability of multilayer feedforward neural networks

Proved that a single hidden layer can approximate any continuous function, providing a theoretical basis for expressive power

Showed that hidden layers learn useful internal representations for a task on their own, making the training of multilayer neural networks practical

Interpreted neural networks as energy-minimizing physical systems and linked them to associative memory and optimization, contributing to the revival of research

Introduced local receptive fields and a hierarchy of simple and complex cells, becoming the direct predecessor of the CNN architecture

Regarded as the first description, predating the 1986 rediscovery. The reference point in the debate over priority for backpropagation

Pinned down the limits mathematically, triggering a slump in neural network research that lasted 10 years or more

The first learning procedure to update weights from errors. Psychological Review 65(6) 386–408

The document that gave the field its name. It led to the workshop the following year, in 1956

Set up a framework for judging machine intelligence by behavior rather than by internal structure. The prototype of today's debates over benchmarks

The prototype of the idea of letting data change the weights. The theoretical root of later learning rules for neural networks

The first attempt to formalize the activity of nerve cells as logical operations, and the mathematical starting point of artificial neural networks