Cross-Entropy And Distillation Loss In KD
This animation compares two loss functions used when training a small student network to mimic a large teacher network. It shows cross-entropy loss measuring the gap between a model's predictions and hard true labels, then contrasts this with distillation loss, which measures the gap between student and teacher soft probability outputs using temperature-scaled softmax. Useful for university students learning how model compression and teacher-student training combine both signals into one weighted objective.
Narrated · 16:9 · Preview before teaching · automatic layout checks do not establish subject accuracy
The prompt that made it
Explain cross-entropy loss and distillation loss in knowledge distillation Audience: freshman in the university
Make the next one in this series
- Explain how temperature scaling smooths softmax outputs in knowledge distillation
- Show how combining hard-label and soft-label losses affects student model accuracy
- Visualize the teacher-student network architecture used in knowledge distillation
