Deep neural models in recent years have been successful in almost every
field, including extremely complex problem statements. However, these models
are huge in size, with millions (and even billions) of parameters, thus
demanding more heavy computation power and failing to be deployed on edge
devices. Besides, the performance boost is highly dependent on redundant
labeled data. To achieve faster speeds and to handle the problems caused by the
lack of data, knowledge distillation (KD) has been proposed to transfer
information learned from one model to another. KD is often characterized by the
so-called `Student-Teacher' (S-T) learning framework and has been broadly
applied in model compression and knowledge transfer. This paper is about KD and
S-T learning, which are being actively studied in recent years. First, we aim
to provide explanations of what KD is and how/why it works. Then, we provide a
comprehensive survey on the recent progress of KD methods together with S-T
frameworks typically for vision tasks. In general, we consider some fundamental
questions that have been driving this research area and thoroughly generalize
the research progress and technical details. Additionally, we systematically
analyze the research status of KD in vision applications. Finally, we discuss
the potentials and open challenges of existing methods and prospect the future
directions of KD and S-T learning.

本文讨论了知识蒸馏和 S-T 学习，提供了对知识蒸馏的解释以及该方法的最新进展、技术细节和视觉应用状况的全面调查，并分析了现有方法的潜力和挑战，展望了知识蒸馏和 S-T 学习的未来方向。