A text-to-speech (TTS) model typically factorizes speech attributes such as
content, speaker and prosody into disentangled representations.Recent works aim
to additionally model the acoustic conditions explicitly, in order to
disentangle the primary speech factors, i.e. linguistic content, prosody and
timbre from any residual factors, such as recording conditions and background
noise.This paper proposes unsupervised, interpretable and fine-grained noise
and prosody modeling. We incorporate adversarial training, representation
bottleneck and utterance-to-frame modeling in order to learn frame-level noise
representations. To the same end, we perform fine-grained prosody modeling via
a Fully Hierarchical Variational AutoEncoder (FVAE) which additionally results
in more expressive speech synthesis.

本文提出了一种基于对抗训练、表示瓶颈和音框级建模的无监督噪音建模方法和基于全层次变分自编码器的细粒度语调建模方法，以更好地进行语音合成并将主要因素与噪声和背景噪声分离。

多说者语音合成的细粒度噪声控制

Fine-grained Noise Control for Multispeaker Speech Synthesis

Custom voice, a specific text to speech (TTS) service in commercial speech
platforms, aims to adapt a source TTS model to synthesize personal voice for a
target speaker using few speech data. Custom voice presents two unique
challenges for TTS adaptation: 1) to support diverse customers, the adaptation
model needs to handle diverse acoustic conditions that could be very different
from source speech data, and 2) to support a large number of customers, the
adaptation parameters need to be small enough for each target speaker to reduce
memory usage while maintaining high voice quality. In this work, we propose
AdaSpeech, an adaptive TTS system for high-quality and efficient customization
of new voices. We design several techniques in AdaSpeech to address the two
challenges in custom voice: 1) To handle different acoustic conditions, we use
two acoustic encoders to extract an utterance-level vector and a sequence of
phoneme-level vectors from the target speech during training; in inference, we
extract the utterance-level vector from a reference speech and use an acoustic
predictor to predict the phoneme-level vectors. 2) To better trade off the
adaptation parameters and voice quality, we introduce conditional layer
normalization in the mel-spectrogram decoder of AdaSpeech, and fine-tune this
part in addition to speaker embedding for adaptation. We pre-train the source
TTS model on LibriTTS datasets and fine-tune it on VCTK and LJSpeech datasets
(with different acoustic conditions from LibriTTS) with few adaptation data,
e.g., 20 sentences, about 1 minute speech. Experiment results show that
AdaSpeech achieves much better adaptation quality than baseline methods, with
only about 5K specific parameters for each speaker, which demonstrates its
effectiveness for custom voice. Audio samples are available at
this https URL

提出了一个自适应的 TTS 系统 AdaSpeech，通过使用两个声学编码器处理不同的声学条件，并在 mel-spectrogram 解码器中引入条件层归一化技术来权衡自适应参数和语音质量，该系统在只有几分钟的训练数据的情况下，对个性化语音化的效果较好，适用于商业语音平台。