We propose a cross-lingual neural codec language model, VALL-E X, for
cross-lingual speech synthesis. Specifically, we extend VALL-E and train a
multi-lingual conditional codec language model to predict the acoustic token
sequences of the target language speech by using both the source language
speech and the target language text as prompts. VALL-E X inherits strong
in-context learning capabilities and can be applied for zero-shot cross-lingual
text-to-speech synthesis and zero-shot speech-to-speech translation tasks.
Experimental results show that it can generate high-quality speech in the
target language via just one speech utterance in the source language as a
prompt while preserving the unseen speaker's voice, emotion, and acoustic
environment. Moreover, VALL-E X effectively alleviates the foreign accent
problems, which can be controlled by a language ID. Audio samples are available
at https://aka.ms/vallex.

本研究提出了一种跨语言神经编解码语言模型，VALL-E X，用于跨语言语音合成，并通过实验证明其在零样本跨语言文本到语音合成和零样本语音到语音翻译任务中能够生成质量高的语音。此外，VALL-E X 可有效缓解外国口音问题，可通过语言 ID 控制。