The Llama 3 Herd of Models published
The Llama 3 Herd of Models published — over 54 days of pretraining on 16K GPUs there were 466 job interruptions in total, 419 of them unexpected, with about 78% attributable to hardware. Even so, effective training time exceeded 90%
Why it matters
The first primary source to publish quantitative data on fault tolerance in large-scale training. Settled, for the whole industry, that checkpointing and automatic recovery are essential
Source — Primary sources first; secondary reports are labelled “Press”
Paperarxiv.orghttps://arxiv.org/abs/2407.21783Added in Week 35, 2026 · See that week