This paper revisits an incredibly simple yet exceedingly effective computing
paradigm, Deep Mutual Learning (DML). We observe that the effectiveness
correlates highly to its excellent generalization quality. In the paper, we
interpret the performance improvement with DML from a novel perspective that it
is roughly an approximate Bayesian posterior sampling procedure. This also
establishes the foundation for applying the Rényi divergence to improve the
original DML, as it brings in the variance control of the prior (in the context
of DML). Therefore, we propose Rényi Divergence Deep Mutual Learning
(RDML). Our empirical results represent the advantage of the marriage of DML
and the Rényi divergence. The flexible control imposed by the Rényi
divergence is able to further improve DML to learn better generalized models.

本文重新审视了一种非常简单却非常有效的计算范例 —— 深度相互学习，并且从一个新颖的角度阐释了 DML 的性能改进。本文提出了一种新的方法 RDML（Rényi Divergence Deep Mutual Learning），证明了 RDML 的优势。

Rényi 离散度深度相互学习

Rényi Divergence Deep Mutual Learning

This paper is the first study to apply deep mutual learning (DML) to
end-to-end ASR models. In DML, multiple models are trained simultaneously and
collaboratively by mimicking each other throughout the training process, which
helps to attain the global optimum and prevent models from making
over-confident predictions. While previous studies applied DML to simple
multi-class classification problems, there are no studies that have used it on
more complex sequence-to-sequence mapping problems. For this reason, this paper
presents a method to apply DML to state-of-the-art Transformer-based end-to-end
ASR models. In particular, we propose to combine DML with recent representative
training techniques. i.e., label smoothing, scheduled sampling, and
SpecAugment, each of which are essential for powerful end-to-end ASR models. We
expect that these training techniques work well with DML because DML has
complementary characteristics. We experimented with two setups for Japanese ASR
tasks: large-scale modeling and compact modeling. We demonstrate that DML
improves the ASR performance of both modeling setups compared with conventional
learning methods including knowledge distillation. We also show that combining
DML with the existing training techniques effectively improves ASR performance.

本论文首次将深度相互学习（DML）应用于端到端自动语音识别模型，通过与标签平滑、定量采样和音频增强等近期代表性训练技术相结合，提高了大规模建模和紧凑建模的日语语音识别任务的性能表现。

深度相互学习的端到端自动语音识别

End-to-End Automatic Speech Recognition with Deep Mutual Learning

Knowledge Distillation (KD) based methods adopt the one-way Knowledge
Transfer (KT) scheme in which training a lower-capacity student network is
guided by a pre-trained high-capacity teacher network. Recently, Deep Mutual
Learning (DML) presented a two-way KT strategy, showing that the student
network can be also helpful to improve the teacher network. In this paper, we
propose Dense Cross-layer Mutual-distillation (DCM), an improved two-way KT
method in which the teacher and student networks are trained collaboratively
from scratch. To augment knowledge representation learning, well-designed
auxiliary classifiers are added to certain hidden layers of both teacher and
student networks. To boost KT performance, we introduce dense bidirectional KD
operations between the layers appended with classifiers. After training, all
auxiliary classifiers are discarded, and thus there are no extra parameters
introduced to final models. We test our method on a variety of KT tasks,
showing its superiorities over related methods. Code is available at
this https URL

在这篇论文中，我们提出了一种改进的双向知识迁移方法 Dense Cross-layer Mutual-distillation (DCM)，它采用了协作训练师生网络的方法，引入了辅助分类器和密集的双向知识蒸馏操作，在各种知识迁移任务中都表现出比相关方法更卓越的性能。

基于稠密交叉层互相蒸馏的知识转移

Knowledge Transfer via Dense Cross-Layer Mutual-Distillation

Model distillation is an effective and widely used technique to transfer
knowledge from a teacher to a student network. The typical application is to
transfer from a powerful large network or ensemble to a small network, that is
better suited to low-memory or fast execution requirements. In this paper, we
present a deep mutual learning (DML) strategy where, rather than one way
transfer between a static pre-defined teacher and a student, an ensemble of
students learn collaboratively and teach each other throughout the training
process. Our experiments show that a variety of network architectures benefit
from mutual learning and achieve compelling results on CIFAR-100 recognition
and Market-1501 person re-identification benchmarks. Surprisingly, it is
revealed that no prior powerful teacher network is necessary -- mutual learning
of a collection of simple student networks works, and moreover outperforms
distillation from a more powerful yet static teacher.

本篇论文提出了深度相互学习策略，其中不是单向从预定义的教师到学生的转移，而是学生网络共同学习和互相教学。通过实验，证明了许多网络架构受益于相互学习，并在 CIFAR-100 识别和 Market-1501 联系人重新识别基准测试中取得了令人信服的结果。