Neural codec language model (LM) has demonstrated strong capability in
zero-shot text-to-speech (TTS) synthesis. However, the codec LM often suffers
from limitations in inference speed and stability, due to its auto-regressive
nature and implicit alignment between text and audio. In this work, to handle
these challenges, we introduce a new variant of neural codec LM, namely TacoLM.
Specifically, TacoLM introduces a gated attention mechanism to improve the
training and inference efficiency and reduce the model size. Meanwhile, an
additional gated cross-attention layer is included for each decoder layer,
which improves the efficiency and content accuracy of the synthesized speech.
In the evaluation of the Librispeech corpus, the proposed TacoLM achieves a
better word error rate, speaker similarity, and mean opinion score, with 90%
fewer parameters and 5.2 times speed up, compared with VALL-E. Demo and code is
available at this https URL

通过引入 TacoLM，一种新型神经编解码语言模型，改进了训练和推理效率，降低模型规模，并通过在每个解码器层引入额外的门控交叉注意力层，提高合成语音的效率和内容准确性。在 Librispeech 资料库的评估中，TacoLM 比 VALL-E 在单词错误率、说话人相似度和平均意见分数方面都取得更好的结果，同时参数减少 90%，提速 5.2 倍。

TacoLM：具有门控注意力的编解码语言模型是高效的零 - shot 文本转语音合成器

TacoLM: GaTed Attention Equipped Codec Language Model are Efficient  Zero-Shot Text to Speech Synthesizers

We introduce a language modeling approach for text to speech synthesis (TTS).
Specifically, we train a neural codec language model (called Vall-E) using
discrete codes derived from an off-the-shelf neural audio codec model, and
regard TTS as a conditional language modeling task rather than continuous
signal regression as in previous work. During the pre-training stage, we scale
up the TTS training data to 60K hours of English speech which is hundreds of
times larger than existing systems. Vall-E emerges in-context learning
capabilities and can be used to synthesize high-quality personalized speech
with only a 3-second enrolled recording of an unseen speaker as an acoustic
prompt. Experiment results show that Vall-E significantly outperforms the
state-of-the-art zero-shot TTS system in terms of speech naturalness and
speaker similarity. In addition, we find Vall-E could preserve the speaker's
emotion and acoustic environment of the acoustic prompt in synthesis. See
this https URL for demos of our work.

本文介绍了一种基于语言模型的文本到语音合成方法，使用名为 Vall-E 的神经编解码器语言模型，通过在预训练阶段将 60K 小时的英语语音数据进行规模扩大，可以使用仅三秒的不同说话人的注册录音作为语音提示来合成高质量的个性化语音，并能够在保持发言人情感和声学环境的前提下，提高声音自然度和发言人相似度。