Research in multilingual speech-to-text translation is topical. Having a
single model that supports multiple translation tasks is desirable. The goal of
this work it to improve cross-lingual transfer learning in multilingual
speech-to-text translation via semantic knowledge distillation. We show that by
initializing the encoder of the encoder-decoder sequence-to-sequence
translation model with SAMU-XLS-R, a multilingual speech transformer encoder
trained using multi-modal (speech-text) semantic knowledge distillation, we
achieve significantly better cross-lingual task knowledge transfer than the
baseline XLS-R, a multilingual speech transformer encoder trained via
self-supervised learning. We demonstrate the effectiveness of our approach on
two popular datasets, namely, CoVoST-2 and Europarl. On the 21 translation
tasks of the CoVoST-2 benchmark, we achieve an average improvement of 12.8 BLEU
points over the baselines. In the zero-shot translation scenario, we achieve an
average gain of 18.8 and 11.9 average BLEU points on unseen medium and
low-resource languages. We make similar observations on Europarl speech
translation benchmark.

本论文旨在通过语义知识蒸馏来改进多语言语音转文本翻译中的跨语言迁移学习。通过使用 SAMU-XLS-R 来初始化编码器，我们在 CoVoST-2 和 Europarl 语音转文本数据集上实现了比基线模型更好的交叉语言任务知识传输，其中平均提高了 12.8 个 BLEU 分数。在零 - shot 翻译情景下，我们在未见过的中低资源语言上平均提高了 18.8 分和 11.9 个 BLEU 分数。