Deep neural networks have achieved remarkable performance for artificial
intelligence tasks. The success behind intelligent systems often relies on
large-scale models with high computational complexity and storage costs. The
over-parameterized networks are often easy to optimize and can achieve better
performance. However, it is challenging to deploy them over resource-limited
edge-devices. Knowledge Distillation (KD) aims to optimize a lightweight
network from the perspective of over-parameterized training. The traditional
offline KD transfers knowledge from a cumbersome teacher to a small and fast
student network. When a sizeable pre-trained teacher network is unavailable,
online KD can improve a group of models by collaborative or mutual learning.
Without needing extra models, Self-KD boosts the network itself using attached
auxiliary architectures. KD mainly involves knowledge extraction and
distillation strategies these two aspects. Beyond KD schemes, various KD
algorithms are widely used in practical applications, such as multi-teacher KD,
cross-modal KD, attention-based KD, data-free KD and adversarial KD. This paper
provides a comprehensive KD survey, including knowledge categories,
distillation schemes and algorithms, as well as some empirical studies on
performance comparison. Finally, we discuss the open challenges of existing KD
works and prospect the future directions.

这篇论文提供了一份全面的知识蒸馏调查，包括知识类别、蒸馏方案和算法，以及一些性能比较的实证研究。