torchft + TorchTitan keep 81.2% efficiency on 300 GPUs with failures every 60 s
With torchft + TorchTitan on 300 GPUs, training efficiency was 81.2% with a failure injected every 60 seconds, and training still converged with failures injected every 15 seconds (1,015 failures). Weights were recovered from peers without reloading checkpoints
Why it matters
An attempt to replace the checkpoint-based recovery paradigm. Treats each step as a distributed transaction
Source — Primary sources first; secondary reports are labelled “Press”
Officialpytorch.orghttps://pytorch.org/blog/fault-tolerant-llama-training-with-2000-synthetic-failures-every-15-seconds-and-no-checkpoints-on-crusoe-l40s/Added in Week 35, 2026 · See that week