This paper presents Translatotron 3, a novel approach to train a direct
speech-to-speech translation model from monolingual speech-text datasets only
in a fully unsupervised manner. Translatotron 3 combines masked autoencoder,
unsupervised embedding mapping, and back-translation to achieve this goal.
Experimental results in speech-to-speech translation tasks between Spanish and
English show that Translatotron 3 outperforms a baseline cascade system,
reporting 18.14 BLEU points improvement on the synthesized
Unpaired-Conversational dataset. In contrast to supervised approaches that
necessitate real paired data, which is unavailable, or specialized modeling to
replicate para-/non-linguistic information, Translatotron 3 showcases its
capability to retain para-/non-linguistic such as pauses, speaking rates, and
speaker identity. Audio samples can be found in our website
this http URL

本文提出了 Translatotron 3，一种无需监督数据集即可以直接语音为输入，并通过采用掩码自编码器、无监督嵌入式映射和回译相结合的方式进行训练的翻译模型。在西班牙语与英语之间的语音到语音翻译任务中，实验结果显示，Translatotron 3 优于基线级联系统，在不配对会话数据集上报告 18.14 个 BLEU 点的改进。与需要现实数据集或专业模型复制非语言信息的监督方法相比，Translatotron 3 展示了保留非语言信息，如停顿、说话速度和说话者身份的能力。