We present a direct speech-to-speech translation (S2ST) model that translates
speech from one language to speech in another language without relying on
intermediate text generation. We tackle the problem by first applying a
self-supervised discrete speech encoder on the target speech and then training
a sequence-to-sequence speech-to-unit translation (S2UT) model to predict the
discrete representations of the target speech. When target text transcripts are
available, we design a joint speech and text training framework that enables
the model to generate dual modality output (speech and text) simultaneously in
the same inference pass. Experiments on the Fisher Spanish-English dataset show
that the proposed framework yields improvement of 6.7 BLEU compared with a
baseline direct S2ST model that predicts spectrogram features. When trained
without any text transcripts, our model performance is comparable to models
that predict spectrograms and are trained with text supervision, showing the
potential of our system for translation between unwritten languages. Audio
samples are available at
this https URL .

本研究提出一种直接语音到语音的翻译模型，它可以在不依赖中间文本生成的情况下将一种语言的语音翻译成另一种语言的语音，并且结合了自我监督离散语音编码和序列到序列的语音到单位翻译，以预测目标语音的离散表示，并在同一推断通过实现同步生成双重模态输出（语音和文本）。在 Fisher 西班牙语 - 英语数据集上，我们的模型较基线直接 S2ST 模型改进了 6.7 BLEU。由于能够支持未书写语言的翻译，当没有文本转录时，我们的模型表现与训练有文本监督的谱图预测模型相当。