Knowledge distillation is a model compression technique in which a smaller network (the student) is trained to reproduce the output distribution of a larger, more capable network (the teacher), rather than learning directly from raw labelled data. Instead of copying hard class labels, the student learns from the teacher's soft probability outputs, which carry richer information about how the teacher relates classes to one another. The result is a compact model that retains much of the teacher's accuracy while being faster and cheaper to run.

Think of it like a senior doctor summarising years of clinical experience into a concise handbook for junior staff. The handbook cannot hold every nuance the doctor carries, but it captures the most useful patterns in a form that a reader can act on quickly.

A practical example is compressing a 70-billion-parameter language model into a 7-billion-parameter model for on-device deployment. During training the student model receives both the ground-truth token and the teacher's full probability vector over the vocabulary. If the teacher assigns 60% probability to "bank" and 30% to "financial institution", the student learns that these outputs are semantically related — a signal a hard label alone would not convey:

loss = alpha * cross_entropy(student_logits, hard_labels)      + (1 - alpha) * kl_divergence(
         softmax(student_logits / T),
         softmax(teacher_logits / T)
       )

Knowledge distillation connects closely to Fine-tuning, since the student is often fine-tuned on teacher signals rather than pre-trained from scratch. It relies on Loss Function design to balance hard-label and soft-label objectives, and relates to Quantization as a complementary compression strategy. Understanding Neural Network architecture choices is important for selecting an appropriate student capacity.

You will encounter knowledge distillation when deploying AI models on mobile devices, edge hardware, or any production environment where latency and memory budgets rule out running the original large model directly.