We focus on the problem of training a deep neural network in generations. The
flowchart is that, in order to optimize the target network (student), another
network (teacher) with the same architecture is first trained, and used to
provide part of supervision signals in the next stage. While this strategy
leads to a higher accuracy, many aspects (e.g., why teacher-student
optimization helps) still need further explorations.
This paper studies this problem from a perspective of controlling the
strictness in training the teacher network. Existing approaches mostly used a
hard distribution (e.g., one-hot vectors) in training, leading to a strict
teacher which itself has a high accuracy, but we argue that the teacher needs
to be more tolerant, although this often implies a lower accuracy. The
implementation is very easy, with merely an extra loss term added to the
teacher network, facilitating a few secondary classes to emerge and complement
to the primary class. Consequently, the teacher provides a milder supervision
signal (a less peaked distribution), and makes it possible for the student to
learn from inter-class similarity and potentially lower the risk of
over-fitting. Experiments are performed on standard image classification tasks
(CIFAR100 and ILSVRC2012). Although the teacher network behaves less powerful,
the students show a persistent ability growth and eventually achieve higher
classification accuracies than other competitors. Model ensemble and transfer
feature extraction also verify the effectiveness of our approach.

本文从严格性控制角度研究了深度神经网络的教师网络和学生网络的优化问题。通过针对教师网络训练中的宽松性进行优化，使得学生网络在学习过程中更容易获得跨类别相似性信息，从而在 CIFAR100 和 ILSVRC2012 等数据集的图像分类任务中取得更高的分类准确率。