Knowledge Distillation
Training a smaller "student" model to imitate the behaviour of a larger "teacher" model, producing a cheaper, faster model that retains much of the teacher's capability.
Knowledge distillation compresses a large model into a smaller one. Rather than training the student only on hard labels — right or wrong — it is trained to match the teacher’s full output distribution, including the relative probabilities the teacher assigns to plausible alternatives. That extra signal, sometimes called “dark knowledge,” teaches the student which mistakes are nearly right and which are not.
Common approaches:
- Response distillation: the student learns from the teacher’s final outputs, often on a large corpus of generated text.
- Feature distillation: the student tries to match the teacher’s internal representations.
- Chain-of-thought distillation: the student is trained on the teacher’s step-by-step reasoning traces, a technique behind several strong small reasoning models.
Distillation is why today’s small models routinely outperform much larger models from a few years ago. It is also a legal and ethical flashpoint: several providers’ terms prohibit training competitors on their outputs, and detecting whether distillation occurred is difficult.