We describe a neural network-based system for text-to-speech (TTS) synthesis
that is able to generate speech audio in the voice of many different speakers,
including those unseen during training. Our system consists of three
independently trained components: (1) a speaker encoder network, trained on a
speaker verification task using an independent dataset of noisy speech from
thousands of speakers without transcripts, to generate a fixed-dimensional
embedding vector from seconds of reference speech from a target speaker; (2) a
sequence-to-sequence synthesis network based on Tacotron 2, which generates a
mel spectrogram from text, conditioned on the speaker embedding; (3) an
auto-regressive WaveNet-based vocoder that converts the mel spectrogram into a
sequence of time domain waveform samples. We demonstrate that the proposed
model is able to transfer the knowledge of speaker variability learned by the
discriminatively-trained speaker encoder to the new task, and is able to
synthesize natural speech from speakers that were not seen during training. We
quantify the importance of training the speaker encoder on a large and diverse
speaker set in order to obtain the best generalization performance. Finally, we
show that randomly sampled speaker embeddings can be used to synthesize speech
in the voice of novel speakers dissimilar from those used in training,
indicating that the model has learned a high quality speaker representation.

描述了一个基于神经网络的文本转语音（TTS）合成系统，可以以许多不同讲话者的声音生成语音音频，该系统由三个独立训练的部分组成，包括训练说话者编码器网络进行讲话者验证任务，基于 Tacotron 2 的序列合成网络，以及将梅尔频谱图转换为时间领域波形样本的自回归 WaveNet 基声码器。