Model distillation, also called knowledge distillation, is a training technique in which a smaller "student" model is trained to replicate the behaviour of a larger "teacher" model by learning from the teacher's output probability distributions rather than from ground-truth labels alone. The teacher's soft probabilities over all possible outputs carry richer information than a hard correct/incorrect signal: they encode the teacher's implicit uncertainty and its assessment of which wrong answers are more plausible than others. The student trains on a combination of this soft target loss and the standard hard-label loss function, often achieving far better performance than a model of the same size trained from scratch.

Think of it like apprenticeship learning: instead of just reading the same textbooks a master craftsperson read, the apprentice watches the master work and learns from the master's moment-to-moment judgements and hesitations — capturing tacit knowledge that the textbooks never articulate.

A minimal distillation training loop computes the KL divergence between teacher and student logits alongside the cross-entropy loss:

import torch.nn.functional as F

def distillation_loss(student_logits, teacher_logits, labels, T=4.0, alpha=0.5):
    soft_loss = F.kl_div(
        F.log_softmax(student_logits / T, dim=-1),
        F.softmax(teacher_logits / T, dim=-1),
        reduction="batchmean"
    ) * (T ** 2)
    hard_loss = F.cross_entropy(student_logits, labels)
    return alpha * soft_loss + (1 - alpha) * hard_loss

DistilBERT, distilled from BERT, retains 97% of BERT's performance at 40% fewer parameters.

Model distillation is closely related to Knowledge Distillation (the general concept), Quantization (another compression approach often combined with distillation), Fine-tuning (the student can be further fine-tuned after distillation), and LoRA (a parameter-efficient alternative for achieving similar compression goals).

You will encounter model distillation whenever deploying large models on resource-constrained hardware — mobile devices, edge servers, or cost-sensitive cloud inference — and in research on making frontier LLM capabilities accessible in smaller, faster packages.