Conversational text-to-speech (TTS) aims to synthesize speech with proper
prosody of reply based on the historical conversation. However, it is still a
challenge to comprehensively model the conversation, and a majority of
conversational TTS systems only focus on extracting global information and omit
local prosody features, which contain important fine-grained information like
keywords and emphasis. Moreover, it is insufficient to only consider the
textual features, and acoustic features also contain various prosody
information. Hence, we propose M2-CTTS, an end-to-end multi-scale multi-modal
conversational text-to-speech system, aiming to comprehensively utilize
historical conversation and enhance prosodic expression. More specifically, we
design a textual context module and an acoustic context module with both
coarse-grained and fine-grained modeling. Experimental results demonstrate that
our model mixed with fine-grained context information and additionally
considering acoustic features achieves better prosody performance and
naturalness in CMOS tests.

提出了一种多尺度，多模态会话文本到语音系统（M2-CTTS），用于综合利用历史会话并增强韵律表达，通过考虑文本和声学因素的粗粒度和细粒度建模，并混合细粒度上下文信息及声学特征，实现了更好的韵律表现和自然度。

M2-CTTS: 端到端的多尺度、多模态会话文本到语音合成

M2-CTTS: End-to-End Multi-scale Multi-modal Conversational  Text-to-Speech Synthesis

Conversational Text-to-Speech (TTS) aims to synthesis an utterance with the
right linguistic and affective prosody in a conversational context. The
correlation between the current utterance and the dialogue history at the
utterance level was used to improve the expressiveness of synthesized speech.
However, the fine-grained information in the dialogue history at the word level
also has an important impact on the prosodic expression of an utterance, which
has not been well studied in the prior work. Therefore, we propose a novel
expressive conversational TTS model, termed as FCTalker, that learn the fine
and coarse grained context dependency at the same time during speech
generation. Specifically, the FCTalker includes fine and coarse grained
encoders to exploit the word and utterance-level context dependency. To model
the word-level dependencies between an utterance and its dialogue history, the
fine-grained dialogue encoder is built on top of a dialogue BERT model. The
experimental results show that the proposed method outperforms all baselines
and generates more expressive speech that is contextually appropriate. We
release the source code at: this https URL.

FCTalker 是一种基于 fine and coarse-grained encoder 的语音合成模型，使用了基于 BERT 的对话编码器来建立话语历史与语音合成之间的单词级别依赖关系，以提高合成语音的表现力和适应性。