Data scarcity and the modality gap between the speech and text modalities are
two major obstacles of end-to-end Speech Translation (ST) systems, thus
hindering their performance. Prior work has attempted to mitigate these
challenges by leveraging external MT data and optimizing distance metrics that
bring closer the speech-text representations. However, achieving competitive
results typically requires some ST data. For this reason, we introduce
ZeroSwot, a method for zero-shot ST that bridges the modality gap without any
paired ST data. Leveraging a novel CTC compression and Optimal Transport, we
train a speech encoder using only ASR data, to align with the representation
space of a massively multilingual MT model. The speech encoder seamlessly
integrates with the MT model at inference, enabling direct translation from
speech to text, across all languages supported by the MT model. Our experiments
show that we can effectively close the modality gap without ST data, while our
results on MuST-C and CoVoST demonstrate our method's superiority over not only
previous zero-shot models, but also supervised ones, achieving state-of-the-art
results.

该论文介绍了一种名为 ZeroSwot 的零样本语音翻译方法，通过利用新颖的 CTC 压缩和最优传输技术，在没有配对的语音翻译数据的情况下，通过仅使用语音识别数据训练语音编码器，实现了语音到文本的直接翻译，展示了该方法在模态差异上的优越性和超越之前的零样本模型以及有监督模型的实验结果，达到了最先进的效果。