Scaling text-to-speech to a large and wild dataset has been proven to be
highly effective in achieving timbre and speech style generalization,
particularly in zero-shot TTS. However, previous works usually encode speech
into latent using audio codec and use autoregressive language models or
diffusion models to generate it, which ignores the intrinsic nature of speech
and may lead to inferior or uncontrollable results. We argue that speech can be
decomposed into several attributes (e.g., content, timbre, prosody, and phase)
and each of them should be modeled using a module with appropriate inductive
biases. From this perspective, we carefully design a novel and large zero-shot
TTS system called Mega-TTS, which is trained with large-scale wild data and
models different attributes in different ways: 1) Instead of using latent
encoded by audio codec as the intermediate feature, we still choose spectrogram
as it separates the phase and other attributes very well. Phase can be
appropriately constructed by the GAN-based vocoder and does not need to be
modeled by the language model. 2) We model the timbre using global vectors
since timbre is a global attribute that changes slowly over time. 3) We further
use a VQGAN-based acoustic model to generate the spectrogram and a latent code
language model to fit the distribution of prosody, since prosody changes
quickly over time in a sentence, and language models can capture both local and
long-range dependencies. We scale Mega-TTS to multi-domain datasets with 20K
hours of speech and evaluate its performance on unseen speakers. Experimental
results demonstrate that Mega-TTS surpasses state-of-the-art TTS systems on
zero-shot TTS, speech editing, and cross-lingual TTS tasks, with superior
naturalness, robustness, and speaker similarity due to the proper inductive
bias of each module. Audio samples are available at
this https URL

该研究基于语音属性的分解和对应的感性偏见，提出了一种基于方谱的 TTS 系统 Mega-TTS，训练了 20k 小时的语音数据，实现了零样本文本到语音的高质量生成。

Mega-TTS: 具有本征归纳偏置的大规模零样本文本转语音

Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive  Bias

We present a novel generative model that combines state-of-the-art neural
text-to-speech (TTS) with semi-supervised probabilistic latent variable models.
By providing partial supervision to some of the latent variables, we are able
to force them to take on consistent and interpretable purposes, which
previously hasn't been possible with purely unsupervised TTS models. We
demonstrate that our model is able to reliably discover and control important
but rarely labelled attributes of speech, such as affect and speaking rate,
with as little as 1% (30 minutes) supervision. Even at such low supervision
levels we do not observe a degradation of synthesis quality compared to a
state-of-the-art baseline. Audio samples are available on the web.

本文提出一种新颖的生成模型，它将最先进的神经文本到语音技术和半监督概率潜变量模型相结合。通过对某些潜变量进行部分监督，我们能够强制它们具有一致和可解释的特征，这在纯无监督的文本到语音模型中过去是不可能的。我们证明了我们的模型能够可靠地发现和控制语音的重要属性（例如情感和语速），即使只监督 1％（30 分钟）。在这样低的监督水平下，我们观察不到合成质量与最先进的基线水平相比的下降。