Distillation (Knowledge Distillation) is a technique where a large, capable model (the “teacher”) trains a smaller model (the “student”) to mimic it, producing a compact model that retains much of the original’s ability.
What it means in plain English
Big models are powerful but expensive to run. Distillation transfers their knowledge into a much smaller model by training the student to reproduce the teacher’s outputs. The student ends up far lighter and faster while keeping a surprising amount of the teacher’s performance. It is a leading way to get near-large-model quality at small-model cost.
Many of the efficient models people run today are distilled from larger ones.
A simple example
A company might take a huge, slow language model and distil it into a smaller one that runs quickly and cheaply in their product — keeping most of the quality while cutting the running cost dramatically.
Why it matters
Distillation is central to making advanced AI affordable and fast enough for real products. It is a major reason capable models can run on modest hardware, and a key technique in the drive toward efficient AI.
Related terms
- Quantization — another model-compression technique.
- Model — both teacher and student are models.
- Inference — distilled models are cheaper to run at inference.
Frequently asked questions
What is model distillation?
It is training a smaller “student” model to reproduce the behaviour of a larger “teacher” model, transferring much of the capability into a more efficient package.
Why distill a model?
A distilled model is smaller, faster, and cheaper to run while keeping much of the larger model’s performance — useful for deployment on limited hardware or at scale.
