Feature-based student-teacher learning, a training method that encourages the student's hidden features to mimic those of the teacher network, is empirically successful in transferring the knowledge from a pre-trained teacher network to the student network. Furthermore, recent empirical results demonstrate that, the teacher's features can boost the student network's generalization even when the student's input sample is corrupted by noise. However, there is a lack of theoretical insights into why and when this method of transferring knowledge can be successful between such heterogeneous tasks. We analyze this method theoretically using deep linear networks, and experimentally using nonlinear networks. We identify three vital factors to the success of the method: (1) whether the student is trained to zero training loss; (2) how knowledgeable the teacher is on the clean-input problem; (3) how the teacher decomposes its knowledge in its hidden features. Lack of proper control in any of the three factors leads to failure of the student-teacher learning method.

该论文探讨了基于特征的学生-教师学习方法的理论解析和实证研究，在深度神经网络中鉴别出对于学生网络成功的三个关键因素，即学生网络是否训练至零误差、教师网络对于原始输入问题的知识水平、以及教师网络如何将其知识分解到其隐藏特征中。

从干净输入到嘈杂输入的师生学习