This paper proposes a neural sequence-to-sequence text-to-speech (TTS) model
which can control latent attributes in the generated speech that are rarely
annotated in the training data, such as speaking style, accent, background
noise, and recording conditions. The model is formulated as a conditional
generative model based on the variational autoencoder (VAE) framework, with two
levels of hierarchical latent variables. The first level is a categorical
variable, which represents attribute groups (e.g. clean/noisy) and provides
interpretability. The second level, conditioned on the first, is a multivariate
Gaussian variable, which characterizes specific attribute configurations (e.g.
noise level, speaking rate) and enables disentangled fine-grained control over
these attributes. This amounts to using a Gaussian mixture model (GMM) for the
latent distribution. Extensive evaluation demonstrates its ability to control
the aforementioned attributes. In particular, we train a high-quality
controllable TTS model on real found data, which is capable of inferring
speaker and style attributes from a noisy utterance and use it to synthesize
clean speech with controllable speaking style.

该研究提出了一种神经序列到序列的文本转语音模型，基于变分自动编码器（VAE）框架，通过两级分层潜变量，实现对生成语音中很少被训练数据标记的潜在属性的控制，包括口音、语速、噪声等，其中第一级为类别变量，第二级为多元高斯变量，通过高斯混合模型 (GMM) 实现。经广泛评估，该模型展现出控制这些潜在属性的能力。