Training a small model to copy what a big model has learned.
Distillation trains a small "student" model to imitate the answers of a large "teacher" model. The student learns not just the right answer but how likely the teacher thought each option was, so it picks things up faster than it would alone. A 2015 paper by Geoffrey Hinton and colleagues made the method widely known.
The small model that comes out is faster and cheaper, so it can run on a phone or laptop, though it is usually a little less capable than its teacher. Collecting another company's AI answers in bulk and distilling them without permission has also led to disputes.
Think of it like this
Imagine borrowing the study notes of the top student in your year. Instead of reading the whole textbook, you follow what they highlighted and where they hesitated, and you catch up much faster. Even so, notes alone rarely make you as good as they are.
For example
If the teacher sees a photo as "70% cat, 25% tiger", the student also learns that the two look quite alike.
Records about this
Source-checked moments from AI history.
The distillation dispute moves from suspicion to quantified accusation. Choosing to disclose its detection work and seek an industry-wide response rather than sue reflects the limits of legal remedies
Formalized distillation, which transfers a large model's knowledge to a small model. The root of today's small-model lineage