Adaptive text to speech (TTS) can synthesize new voices in zero-shot
scenarios efficiently, by using a well-trained source TTS model without
adapting it on the speech data of new speakers. Considering seen and unseen
speakers have diverse characteristics, zero-shot adaptive TTS requires strong
generalization ability on speaker characteristics, which brings modeling
challenges. In this paper, we develop AdaSpeech 4, a zero-shot adaptive TTS
system for high-quality speech synthesis. We model the speaker characteristics
systematically to improve the generalization on new speakers. Generally, the
modeling of speaker characteristics can be categorized into three steps:
extracting speaker representation, taking this speaker representation as
condition, and synthesizing speech/mel-spectrogram given this speaker
representation. Accordingly, we improve the modeling in three steps: 1) To
extract speaker representation with better generalization, we factorize the
speaker characteristics into basis vectors and extract speaker representation
by weighted combining of these basis vectors through attention. 2) We leverage
conditional layer normalization to integrate the extracted speaker
representation to TTS model. 3) We propose a novel supervision loss based on
the distribution of basis vectors to maintain the corresponding speaker
characteristics in generated mel-spectrograms. Without any fine-tuning,
AdaSpeech 4 achieves better voice quality and similarity than baselines in
multiple datasets.

本文提出了一个高质量的语音合成系统 AdaSpeech 4，用于零样本适应性文本到语音。该系统通过将说话者特征系统地建模来提高对新说话者的泛化能力，并通过三个步骤改善了建模。除了利用条件层归一化将提取的说话人特征整合到 TTS 模型中之外，本文基于基向量的分布提出了一种新的监督损失，以在生成的 Mel 声谱图中保持相应的说话人特征。在多个数据集中，AdaSpeech 4 实现了比基线更好的语音质量和相似性，而无需进行任何微调。