Joint speech-language training is challenging due to the large demand for
training data and GPU consumption, as well as the modality gap between speech
and language. We present ComSL, a speech-language model built atop a composite
architecture of public pretrained speech-only and language-only models and
optimized data-efficiently for spoken language tasks. Particularly, we propose
to incorporate cross-modality learning into transfer learning and conduct them
simultaneously for downstream tasks in a multi-task learning manner. Our
approach has demonstrated effectiveness in end-to-end speech-to-text
translation tasks, achieving a new state-of-the-art average BLEU score of 31.5
on the multilingual speech to English text translation task for 21 languages,
as measured on the public CoVoST2 evaluation set.

通过 ComSL 模型的建立，我们将跨模态学习和迁移学习相结合，在多任务学习的框架下进行，有效地实现了端到端的语音转文本翻译任务。在 21 种语言的 CoVoST2 公共数据集的评估中，我们的方法在多语言语音到英文文本翻译任务中实现了新的最高 BLEU 评分平均值 31.5。