End-to-end Speech Translation (ST) aims at translating the source language
speech into target language text without generating the intermediate
transcriptions. However, the training of end-to-end methods relies on parallel
ST data, which are difficult and expensive to obtain. Fortunately, the
supervised data for automatic speech recognition (ASR) and machine translation
(MT) are usually more accessible, making zero-shot speech translation a
potential direction. Existing zero-shot methods fail to align the two
modalities of speech and text into a shared semantic space, resulting in much
worse performance compared to the supervised ST methods. In order to enable
zero-shot ST, we propose a novel Discrete Cross-Modal Alignment (DCMA) method
that employs a shared discrete vocabulary space to accommodate and match both
modalities of speech and text. Specifically, we introduce a vector quantization
module to discretize the continuous representations of speech and text into a
finite set of virtual tokens, and use ASR data to map corresponding speech and
text to the same virtual token in a shared codebook. This way, source language
speech can be embedded in the same semantic space as the source language text,
which can be then transformed into target language text with an MT module.
Experiments on multiple language pairs demonstrate that our zero-shot ST method
significantly improves the SOTA, and even performers on par with the strong
supervised ST baselines.

该研究提出了一种新的离线语音翻译方法，称为离散跨模态对齐方法（Discrete Cross-Modal Alignment method），该方法能有效地将语音和文字的两种模态映射到共享的语义空间中，从而提高零 - shot 语音翻译的性能。