The performance of a distillation-based compressed network is governed by the
quality of distillation. The reason for the suboptimal distillation of a large
network (teacher) to a smaller network (student) is largely attributed to the
gap in the learning capacities of given teacher-student pair. While it is hard
to distill all the knowledge of a teacher, the quality of distillation can be
controlled to a large extent to achieve better performance. Our experiments
show that the quality of distillation is largely governed by the quality of
teacher's response, which in turn is heavily affected by the presence of
similarity information in its response. A well-trained large capacity teacher
loses similarity information between classes in the process of learning
fine-grained discriminative properties for classification. The absence of
similarity information causes the distillation process to be reduced from one
example-many class learning to one example-one class learning, thereby
throttling the flow of diverse knowledge from the teacher. With the implicit
assumption that only the instilled knowledge can be distilled, instead of
focusing only on the knowledge distilling process, we scrutinize the knowledge
inculcation process. We argue that for a given teacher-student pair, the
quality of distillation can be improved by finding the sweet spot between batch
size and number of epochs while training the teacher. We discuss the steps to
find this sweet spot for better distillation. We also propose the distillation
hypothesis to differentiate the behavior of the distillation process between
knowledge distillation and regularization effect. We conduct all our
experiments on three different datasets.

本文探讨了如何提高知识压缩的效果，发现教师神经网络的反应质量和相似性信息密切相关，而采用适当的批量和历元数训练教师可以提高知识压缩的效果。