Knowledge Distillation (KD) aims to distill the knowledge of a cumbersome
teacher model into a lightweight student model. Its success is generally
attributed to the privileged information on similarities among categories
provided by the teacher model, and in this sense, only strong teacher models
are deployed to teach weaker students in practice. In this work, we challenge
this common belief by following experimental observations: 1) beyond the
acknowledgment that the teacher can improve the student, the student can also
enhance the teacher significantly by reversing the KD procedure; 2) a
poorly-trained teacher with much lower accuracy than the student can still
improve the latter significantly. To explain these observations, we provide a
theoretical analysis of the relationships between KD and label smoothing
regularization. We prove that 1) KD is a type of learned label smoothing
regularization and 2) label smoothing regularization provides a virtual teacher
model for KD. From these results, we argue that the success of KD is not fully
due to the similarity information between categories from teachers, but also to
the regularization of soft targets, which is equally or even more important.
Based on these analyses, we further propose a novel Teacher-free Knowledge
Distillation (Tf-KD) framework, where a student model learns from itself or
manuallydesigned regularization distribution. The Tf-KD achieves comparable
performance with normal KD from a superior teacher, which is well applied when
a stronger teacher model is unavailable. Meanwhile, Tf-KD is generic and can be
directly deployed for training deep neural networks. Without any extra
computation cost, Tf-KD achieves up to 0.65\% improvement on ImageNet over
well-established baseline models, which is superior to label smoothing
regularization.

本文挑战了使用强大的 teacher model 教授弱 student model 的共识，详细阐述了 KD 和 label smoothing regularization 之间的关系，提出了 Teacher-free Knowledge Distillation（Tf-KD）框架，通过自学和手动设计正则化分布，实现了与正常 KD 相媲美的性能。

通过标签平滑正则化重新审视知识蒸馏

Revisiting Knowledge Distillation via Label Smoothing Regularization

Targeted sentiment classification aims at determining the sentimental
tendency towards specific targets. Most of the previous approaches model
context and target words with RNN and attention. However, RNNs are difficult to
parallelize and truncated backpropagation through time brings difficulty in
remembering long-term patterns. To address this issue, this paper proposes an
Attentional Encoder Network (AEN) which eschews recurrence and employs
attention based encoders for the modeling between context and target. We raise
the label unreliability issue and introduce label smoothing regularization. We
also apply pre-trained BERT to this task and obtain new state-of-the-art
results. Experiments and analysis demonstrate the effectiveness and lightweight
of our model.

本文提出了一个名为 Attentional Encoder Network (AEN) 的网络，它使用基于注意力机制的编码器来处理上下文与目标之间的建模，采用标签平滑正则化来解决标签不可靠的问题，并将预训练的 BERT 应用于情感分类任务，实现了新的最先进的结果。实验和分析证明了我们模型的有效性和轻量级。