NVIDIA presents Megatron-LM
Presented Megatron-LM — implemented intra-layer tensor parallelism in PyTorch alone. Sustained 15.1 PFLOPS with an 8.3B model on 512 GPUs
Why it matters
The standard implementation of large-scale model parallelism
Source — Primary sources first; secondary reports are labelled “Press”
Paperarxiv.orghttps://arxiv.org/abs/1909.08053Added in Week 35, 2026 · See that week