Modern text-to-speech (TTS) systems are able to generate audio that sounds
almost as natural as human speech. However, the bar of developing high-quality
TTS systems remains high since a sizable set of studio-quality <text, audio>
pairs is usually required. Compared to commercial data used to develop
state-of-the-art systems, publicly available data are usually worse in terms of
both quality and size. Audio generated by TTS systems trained on publicly
available data tends to not only sound less natural, but also exhibits more
background noise. In this work, we aim to lower TTS systems' reliance on
high-quality data by providing them the textual knowledge extracted by deep
pre-trained language models during training. In particular, we investigate the
use of BERT to assist the training of Tacotron-2, a state of the art TTS
consisting of an encoder and an attention-based decoder. BERT representations
learned from large amounts of unlabeled text data are shown to contain very
rich semantic and syntactic information about the input text, and have
potential to be leveraged by a TTS system to compensate the lack of
high-quality data. We incorporate BERT as a parallel branch to the Tacotron-2
encoder with its own attention head. For an input text, it is simultaneously
passed into BERT and the Tacotron-2 encoder. The representations extracted by
the two branches are concatenated and then fed to the decoder. As a preliminary
study, although we have not found incorporating BERT into Tacotron-2 generates
more natural or cleaner speech at a human-perceivable level, we observe
improvements in other aspects such as the model is being significantly better
at knowing when to stop decoding such that there is much less babbling at the
end of the synthesized audio and faster convergence during training.

本论文研究了通过引入 BERT 模型辅助训练 TTS 模型 Tacotron-2，以缓解高质量语音库的不足问题，并在模型训练中观察到模型收敛更快、结果中无杂音等优点。