In this paper, we propose a method to learn unified representations of
multilingual speech and text with a single model, especially focusing on the
purpose of speech synthesis. We represent multilingual speech audio with speech
units, the quantized representations of speech features encoded from a
self-supervised speech model. Therefore, we can focus on their linguistic
content by treating the audio as pseudo text and can build a unified
representation of speech and text. Then, we propose to train an encoder-decoder
structured model with a Unit-to-Unit Translation (UTUT) objective on
multilingual data. Specifically, by conditioning the encoder with the source
language token and the decoder with the target language token, the model is
optimized to translate the spoken language into that of the target language, in
a many-to-many language translation setting. Therefore, the model can build the
knowledge of how spoken languages are comprehended and how to relate them to
different languages. A single pre-trained model with UTUT can be employed for
diverse multilingual speech- and text-related tasks, such as Speech-to-Speech
Translation (STS), multilingual Text-to-Speech Synthesis (TTS), and
Text-to-Speech Translation (TTST). By conducting comprehensive experiments
encompassing various languages, we validate the efficacy of the proposed method
across diverse multilingual tasks. Moreover, we show UTUT can perform
many-to-many language STS, which has not been previously explored in the
literature. Samples are available on this https URL

我们提出了一种方法，通过单一模型学习多语言语音和文本的统一表示，特别关注语音合成的目的。我们使用语音单元表示多语言语音音频，这是从自我监督语音模型编码的语音特征的量化表示。因此，我们可以将语音视为伪文本，并构建语音和文本的统一表示。然后，我们提出在多语言数据上通过单位到单位翻译 (UTUT) 目标训练编码器 - 解码器结构模型。通过使用源语言标记来确定编码器，使用目标语言标记来确定解码器，该模型被优化为将口语语言翻译为目标语言的翻译模型。因此，该模型可以建立口语语言的理解方式，并将其与不同语言相关联。单个经过预训练的 UTUT 模型可以用于各种多语言语音和文本相关任务，如语音到语音翻译 (STS)、多语言文本到语音合成 (TTS) 和文本到语音翻译 (TTST)。我们通过包含各种语言的综合实验验证了所提出方法的有效性。此外，我们展示了 UTUT 可以执行先前在文献中未探索的多对多语言 STS。样例可在此网址获取：https://URL