Recent advancements in neural end-to-end TTS models have shown high-quality,
natural synthesized speech in a conventional sentence-based TTS. However, it is
still challenging to reproduce similar high quality when a whole paragraph is
considered in TTS, where a large amount of contextual information needs to be
considered in building a paragraph-based TTS model. To alleviate the difficulty
in training, we propose to model linguistic and prosodic information by
considering cross-sentence, embedded structure in training. Three sub-modules,
including linguistics-aware, prosody-aware and sentence-position networks, are
trained together with a modified Tacotron2. Specifically, to learn the
information embedded in a paragraph and the relations among the corresponding
component sentences, we utilize linguistics-aware and prosody-aware networks.
The information in a paragraph is captured by encoders and the inter-sentence
information in a paragraph is learned with multi-head attention mechanisms. The
relative sentence position in a paragraph is explicitly exploited by a
sentence-position network. Trained on a storytelling audio-book corpus (4.08
hours), recorded by a female Mandarin Chinese speaker, the proposed TTS model
demonstrates that it can produce rather natural and good-quality speech
paragraph-wise. The cross-sentence contextual information, such as break and
prosodic variations between consecutive sentences, can be better predicted and
rendered than the sentence-based model. Tested on paragraph texts, of which the
lengths are similar to, longer than, or much longer than the typical paragraph
length of the training data, the TTS speech produced by the new model is
consistently preferred over the sentence-based model in subjective tests and
confirmed in objective measures.

本文提出了一种基于跨句嵌入结构的 TTS 模型，包含语言学信息，韵律信息和位置信息，并在女性普通话语音书籍语料库上进行了训练，结果表明该模型能够生成自然、优质的段落朗读文本，且在主观和客观测试中都表现优于基于句子的模型。

ParaTTS: 基于段落的语音合成中学习跨句子的语言和韵律信息

ParaTTS: Learning Linguistic and Prosodic Cross-sentence Information in Paragraph-based TTS

Neural end-to-end TTS can generate very high-quality synthesized speech, and
even close to human recording within similar domain text. However, it performs
unsatisfactory when scaling it to challenging test sets. One concern is that
the encoder-decoder with attention-based network adopts autoregressive
generative sequence model with the limitation of "exposure bias" To address
this issue, we propose two novel methods, which learn to predict future by
improving agreement between forward and backward decoding sequence. The first
one is achieved by introducing divergence regularization terms into model
training objective to reduce the mismatch between two directional models,
namely L2R and R2L (which generates targets from left-to-right and
right-to-left, respectively). While the second one operates on decoder-level
and exploits the future information during decoding. In addition, we employ a
joint training strategy to allow forward and backward decoding to improve each
other in an interactive process. Experimental results show our proposed methods
especially the second one (bidirectional decoder regularization), leads a
significantly improvement on both robustness and overall naturalness, as
outperforming baseline (the revised version of Tacotron2) with a MOS gap of
0.14 in a challenging test, and achieving close to human quality (4.42 vs. 4.49
in MOS) on general test.

该研究提出了两种新方法以解决神经端到端 TTS 模型在挑战性测试集中表现不佳的问题，包括引入差异正则化项和双向解码器正则化，并采用联合训练策略，在提高鲁棒性和自然度方面均取得了显著的改进。