Recent neural speech synthesis systems have gradually focused on the control
of prosody to improve the quality of synthesized speech, but they rarely
consider the variability of prosody and the correlation between prosody and
semantics together. In this paper, a prosody learning mechanism is proposed to
model the prosody of speech based on TTS system, where the prosody information
of speech is extracted from the melspectrum by a prosody learner and combined
with the phoneme sequence to reconstruct the mel-spectrum. Meanwhile, the
sematic features of text from the pre-trained language model is introduced to
improve the prosody prediction results. In addition, a novel self-attention
structure, named as local attention, is proposed to lift this restriction of
input text length, where the relative position information of the sequence is
modeled by the relative position matrices so that the position encodings is no
longer needed. Experiments on English and Mandarin show that speech with more
satisfactory prosody has obtained in our model. Especially in Mandarin
synthesis, our proposed model outperforms baseline model with a MOS gap of
0.08, and the overall naturalness of the synthesized speech has been
significantly improved.

本文提出了一种基于 TTS 系统的韵律学习机制，利用本文语言模型的语义特征改进韵律预测结果，并提出了一种名为局部注意力的自注意力结构，以消除输入文本长度局限性，实验证明，我们的模型在英语和汉语合成中产生了更加令人满意的韵律，尤其在汉语合成中，我们的模型优于基准模型，MOS 差距为 0.08，合成语音的整体自然度得到了显著提高。