Large-scale self-supervised pre-trained speech encoders outperform
conventional approaches in speech recognition and translation tasks. Due to the
high cost of developing these large models, building new encoders for new tasks
and deploying them to on-device applications are infeasible. Prior studies
propose model compression methods to address this issue, but those works focus
on smaller models and less realistic tasks. Thus, we propose Contrastive
Layer-to-layer Distillation (CoLLD), a novel knowledge distillation method to
compress pre-trained speech encoders by leveraging masked prediction and
contrastive learning to train student models to copy the behavior of a large
teacher model. CoLLD outperforms prior methods and closes the gap between small
and large models on multilingual speech-to-text translation and recognition
benchmarks.

大规模无监督预训练语音编码器优于传统方法在语音识别和翻译任务中。然而，由于开发这些大型模型的高成本，构建新的编码器用于新任务并将其部署到设备应用上是不可行的。因此，先前的研究提出了模型压缩方法来解决该问题，但这些方法主要关注较小的模型和较不现实的任务。因此，我们提出了一种新的知识蒸馏方法，即对比层间蒸馏（CoLLD），通过利用掩码预测和对比学习来训练学生模型以模仿大型教师模型的行为，从而压缩预训练语音编码器。CoLLD 在多语言语音到文本翻译和识别基准测试中优于先前的方法，缩小了小型和大型模型之间的差距。

CoLLD: 对比层间蒸馏用于压缩多语言预训练语音编码器

CoLLD: Contrastive Layer-to-layer Distillation for Compressing  Multilingual Pre-trained Speech Encoders

Research in multilingual speech-to-text translation is topical. Having a
single model that supports multiple translation tasks is desirable. The goal of
this work it to improve cross-lingual transfer learning in multilingual
speech-to-text translation via semantic knowledge distillation. We show that by
initializing the encoder of the encoder-decoder sequence-to-sequence
translation model with SAMU-XLS-R, a multilingual speech transformer encoder
trained using multi-modal (speech-text) semantic knowledge distillation, we
achieve significantly better cross-lingual task knowledge transfer than the
baseline XLS-R, a multilingual speech transformer encoder trained via
self-supervised learning. We demonstrate the effectiveness of our approach on
two popular datasets, namely, CoVoST-2 and Europarl. On the 21 translation
tasks of the CoVoST-2 benchmark, we achieve an average improvement of 12.8 BLEU
points over the baselines. In the zero-shot translation scenario, we achieve an
average gain of 18.8 and 11.9 average BLEU points on unseen medium and
low-resource languages. We make similar observations on Europarl speech
translation benchmark.

本论文旨在通过语义知识蒸馏来改进多语言语音转文本翻译中的跨语言迁移学习。通过使用 SAMU-XLS-R 来初始化编码器，我们在 CoVoST-2 和 Europarl 语音转文本数据集上实现了比基线模型更好的交叉语言任务知识传输，其中平均提高了 12.8 个 BLEU 分数。在零 - shot 翻译情景下，我们在未见过的中低资源语言上平均提高了 18.8 分和 11.9 个 BLEU 分数。