We propose a Multi-Stage, Multi-Codebook (MSMC) approach to high-performance
neural TTS synthesis. A vector-quantized, variational autoencoder (VQ-VAE)
based feature analyzer is used to encode Mel spectrograms of speech training
data by down-sampling progressively in multiple stages into MSMC
Representations (MSMCRs) with different time resolutions, and quantizing them
with multiple VQ codebooks, respectively. Multi-stage predictors are trained to
map the input text sequence to MSMCRs progressively by minimizing a combined
loss of the reconstruction Mean Square Error (MSE) and "triplet loss". In
synthesis, the neural vocoder converts the predicted MSMCRs into final speech
waveforms. The proposed approach is trained and tested with an English TTS
database of 16 hours by a female speaker. The proposed TTS achieves an MOS
score of 4.41, which outperforms the baseline with an MOS of 3.62. Compact
versions of the proposed TTS with much less parameters can still preserve high
MOS scores. Ablation studies show that both multiple stages and multiple
codebooks are effective for achieving high TTS performance.

提出一种多阶段、多码本（MSMC）方法来高效实现神经 TTS 合成，该方法使用基于向量量化的变分自编码器（VQ-VAE）编码语音训练数据的 Mel 频谱图，并通过多阶段逐渐进行下采样，将它们量化为具有不同时间分辨率的多个 MSMC 表示，同时使用多个 VQ 码本。在合成中，神经声码器将预测的 MSMCR 转换为最终语音波形，该方法在 16 小时的英语 TTS 数据库中，以女性说话者的 TTS 得分（MOS）为 4.41，比基准值的 MOS（3.62）更出色，同时低参数的紧凑版本仍然可以保持高 MOS 得分，消融研究表明，多个阶段和多个码本对于实现高性能的 TTS 都是有效的。