End-to-end speech translation (ST) is the task of translating speech signals
in the source language into text in the target language. As a cross-modal task,
end-to-end ST is difficult to train with limited data. Existing methods often
try to transfer knowledge from machine translation (MT), but their performances
are restricted by the modality gap between speech and text. In this paper, we
propose Cross-modal Mixup via Optimal Transport CMOT to overcome the modality
gap. We find the alignment between speech and text sequences via optimal
transport and then mix up the sequences from different modalities at a token
level using the alignment. Experiments on the MuST-C ST benchmark demonstrate
that CMOT achieves an average BLEU of 30.0 in 8 translation directions,
outperforming previous methods. Further analysis shows CMOT can adaptively find
the alignment between modalities, which helps alleviate the modality gap
between speech and text. Code is publicly available at
this https URL

该论文研究了语音翻译的跨模态任务，以及如何通过使用最优传输和跨模态混合等方法，来克服语音和文本之间的模态差异，从而提高翻译的准确性。实验结果表明，该方法比以往方法表现更优秀。