A chip that does huge numbers of simple calculations at once: the workhorse of AI.
A GPU (graphics processing unit) was built to colour in millions of pixels on a game screen at the same time. It packs thousands of small calculating units, so it's great at splitting easy maths into pieces and doing them all at once. Training AI turns out to be exactly that kind of work: endless simple multiplying and adding.
From the mid-2000s, studies showed GPUs could train neural networks much faster, and the 2012 ImageNet winners used two graphics cards. Today's large models train for weeks or months on thousands or even tens of thousands of GPUs or similar AI chips in data centers. That has made GPUs expensive and hard to get.
Think of it like this
Say you have 10,000 maths problems. A few brilliant students working through them one by one is slow; a few thousand students each taking a handful at the same time is far faster. A CPU is like the few brilliant students; a GPU is the crowd of thousands.
For example
Meta reported linking up to 16,000 GPUs together to train Llama 3.
Records about this
Source-checked moments from AI history.
Established in a paper the combination of GPUs, large-scale data and deep CNNs. The conventional starting point of the current deep learning era
An early case of applying GPUs to deep learning training in earnest. The starting point of the premise that compute is a resource
Lowered the barrier to general-purpose computing on GPUs — the software foundation that all of deep learning later rode on