Model distillation is an effective and widely used technique to transfer
knowledge from a teacher to a student network. The typical application is to
transfer from a powerful large network or ensemble to a small network, that is
better suited to low-memory or fast execution requirements. In this paper, we
present a deep mutual learning (DML) strategy where, rather than one way
transfer between a static pre-defined teacher and a student, an ensemble of
students learn collaboratively and teach each other throughout the training
process. Our experiments show that a variety of network architectures benefit
from mutual learning and achieve compelling results on CIFAR-100 recognition
and Market-1501 person re-identification benchmarks. Surprisingly, it is
revealed that no prior powerful teacher network is necessary -- mutual learning
of a collection of simple student networks works, and moreover outperforms
distillation from a more powerful yet static teacher.

本篇论文提出了深度相互学习策略，其中不是单向从预定义的教师到学生的转移，而是学生网络共同学习和互相教学。通过实验，证明了许多网络架构受益于相互学习，并在 CIFAR-100 识别和 Market-1501 联系人重新识别基准测试中取得了令人信服的结果。

深度相互学习

Deep Mutual Learning

Cross-lingual text classification(CLTC) is the task of classifying documents
written in different languages into the same taxonomy of categories. This paper
presents a novel approach to CLTC that builds on model distillation, which
adapts and extends a framework originally proposed for model compression. Using
soft probabilistic predictions for the documents in a label-rich language as
the (induced) supervisory labels in a parallel corpus of documents, we train
classifiers successfully for new languages in which labeled training data are
not available. An adversarial feature adaptation technique is also applied
during the model training to reduce distribution mismatch. We conducted
experiments on two benchmark CLTC datasets, treating English as the source
language and German, French, Japan and Chinese as the unlabeled target
languages. The proposed approach had the advantageous or comparable performance
of the other state-of-art methods.

本研究提出了一个基于模型蒸馏的跨语言文本分类新方法，使用软概率预测作为并行语料库中标签丰富语言中的监督标签，成功地对目标语言训练分类器，并应用对抗式特征适应技术以减少分布不匹配现象，实验证明该方法具有优越的性能。