We introduce a language modeling approach for text to speech synthesis (TTS).
Specifically, we train a neural codec language model (called Vall-E) using
discrete codes derived from an off-the-shelf neural audio codec model, and
regard TTS as a conditional language modeling task rather than continuous
signal regression as in previous work. During the pre-training stage, we scale
up the TTS training data to 60K hours of English speech which is hundreds of
times larger than existing systems. Vall-E emerges in-context learning
capabilities and can be used to synthesize high-quality personalized speech
with only a 3-second enrolled recording of an unseen speaker as an acoustic
prompt. Experiment results show that Vall-E significantly outperforms the
state-of-the-art zero-shot TTS system in terms of speech naturalness and
speaker similarity. In addition, we find Vall-E could preserve the speaker's
emotion and acoustic environment of the acoustic prompt in synthesis. See
this https URL for demos of our work.

本文介绍了一种基于语言模型的文本到语音合成方法，使用名为 Vall-E 的神经编解码器语言模型，通过在预训练阶段将 60K 小时的英语语音数据进行规模扩大，可以使用仅三秒的不同说话人的注册录音作为语音提示来合成高质量的个性化语音，并能够在保持发言人情感和声学环境的前提下，提高声音自然度和发言人相似度。