2018, in 7 pieces

Fine-tuned with just one additional output layer, it set new state-of-the-art results on 11 NLP tasks, including GLUE 80.4% and SQuAD v1.1 F1 93.2 (as reported by the authors)

Reported that combining unsupervised generative pre-training with supervised fine-tuning improved the state of the art on 9 of the 12 tasks studied

Defined language understanding as a bundle of multiple tasks rather than a single task. The prototype of every later LLM leaderboard

The layer that made the inside of a node one big GPU. The physical precondition for tensor parallelism becoming practical

The turning point that brought distributed training down from research infrastructure to an everyday tool

The first practical compiler alternative to the monopoly of vendor kernel libraries. Brought the concept of automatic scheduling to the industry

With composable grad/jit/vmap transformations and XLA compilation, it formed the standard route for large-scale training on TPUs. The repository states that it is a research project, not an official Google product