Benefiting from the sequence-level knowledge distillation, the
Non-Autoregressive Transformer (NAT) achieves great success in neural machine
translation tasks. However, existing knowledge distillation has side effects,
such as propagating errors from the teacher to NAT students, which may limit
further improvements of NAT models and are rarely discussed in existing
research. In this paper, we introduce selective knowledge distillation by
introducing an NAT evaluator to select NAT-friendly targets that are of high
quality and easy to learn. In addition, we introduce a simple yet effective
progressive distillation method to boost NAT performance. Experiment results on
multiple WMT language directions and several representative NAT models show
that our approach can realize a flexible trade-off between the quality and
complexity of training data for NAT models, achieving strong performances.
Further analysis shows that distilling only 5% of the raw translations can help
an NAT outperform its counterpart trained on raw data by about 2.4 BLEU.

本文提出了一种选择性知识蒸馏技术，通过 NAT 评估器选择高质量且易于学习的 NAT 友好目标，并引入简单而有效的渐进式蒸馏方法来提高 NAT 性能，在多个 WMT 语言方向和多个代表性 NAT 模型上进行实验。结果表明，我们的方法可以实现训练数据的质量和复杂性之间的灵活权衡，取得了出色的性能。进一步的分析表明，只通过 5% 的原始翻译数据，就可以使 NAT 在 BLEU 值上比原始数据训练的对手高出约 2.4 BLEU。

非自回归神经机器翻译的选择性知识蒸馏

Selective Knowledge Distillation for Non-Autoregressive Neural Machine  Translation

Typically, the deployment of face recognition models in the wild needs to
identify low-resolution faces with extremely low computational cost. To address
this problem, a feasible solution is compressing a complex face model to
achieve higher speed and lower memory at the cost of minimal performance drop.
Inspired by that, this paper proposes a learning approach to recognize
low-resolution faces via selective knowledge distillation. In this approach, a
two-stream convolutional neural network (CNN) is first initialized to recognize
high-resolution faces and resolution-degraded faces with a teacher stream and a
student stream, respectively. The teacher stream is represented by a complex
CNN for high-accuracy recognition, and the student stream is represented by a
much simpler CNN for low-complexity recognition. To avoid significant
performance drop at the student stream, we then selectively distil the most
informative facial features from the teacher stream by solving a sparse graph
optimization problem, which are then used to regularize the fine-tuning process
of the student stream. In this way, the student stream is actually trained by
simultaneously handling two tasks with limited computational resources:
approximating the most informative facial cues via feature regression, and
recovering the missing facial cues via low-resolution face classification.
Experimental results show that the student stream performs impressively in
recognizing low-resolution faces and costs only 0.15MB memory and runs at 418
faces per second on CPU and 9,433 faces per second on GPU.

本研究提出了一种使用选择性知识蒸馏学习低分辨率人脸识别的方法，在损失性能的同时实现更高速度和更低内存消耗，通过优化图问题，选择性地从一个复杂的面部模型中提取最具信息量的面部特征来让一个简单的卷积神经网络完成相应识别任务。