In speech translation, leveraging multimodal data to improve model
performance and address limitations of individual modalities has shown
significant effectiveness. In this paper, we harness the complementary
strengths of speech and text, which are disparate modalities. We observe three
levels of modality gap between them, denoted by Modal input representation,
Modal semantic, and Modal hidden states. To tackle these gaps, we propose
\textbf{F}use-\textbf{S}peech-\textbf{T}ext (\textbf{FST}), a cross-modal model
which supports three distinct input modalities for translation: speech, text,
and fused speech-text. We leverage multiple techniques for cross-modal
alignment and conduct a comprehensive analysis to assess its impact on speech
translation, machine translation, and fused speech-text translation. We
evaluate FST on MuST-C, GigaST, and newstest benchmark. Experiments show that
the proposed FST achieves an average 34.0 BLEU on MuST-C
En$\rightarrow$De/Es/Fr (vs SOTA +1.1 BLEU). Further experiments demonstrate
that FST does not degrade on MT task, as observed in prior works. Instead, it
yields an average improvement of 3.2 BLEU over the pre-trained MT model.

该研究提出了 FST 跨模态模型，支持语音、文本和语音文本融合三种不同类型的输入模态进行翻译，技术包括多模态对齐等方法，实验结果表明该模型在多项测试中比传统方法有显著提升，尤其是在语音翻译领域。