Expressive Text-to-Speech (TTS) using reference speech has been studied
extensively to synthesize natural speech, but there are limitations to
obtaining well-represented styles and improving model generalization ability.
In this study, we present Diffusion-based EXpressive TTS (DEX-TTS), an acoustic
model designed for reference-based speech synthesis with enhanced style
representations. Based on a general diffusion TTS framework, DEX-TTS includes
encoders and adapters to handle styles extracted from reference speech. Key
innovations contain the differentiation of styles into time-invariant and
time-variant categories for effective style extraction, as well as the design
of encoders and adapters with high generalization ability. In addition, we
introduce overlapping patchify and convolution-frequency patch embedding
strategies to improve DiT-based diffusion networks for TTS. DEX-TTS yields
outstanding performance in terms of objective and subjective evaluation in
English multi-speaker and emotional multi-speaker datasets, without relying on
pre-training strategies. Lastly, the comparison results for the general TTS on
a single-speaker dataset verify the effectiveness of our enhanced diffusion
backbone. Demos are available here.

基于参考语音的语音合成已经被广泛研究来合成自然语音，但在获取良好的表达风格和提高模型泛化能力方面存在限制。本研究提出了基于扩散的表达性语音合成 (DEX-TTS)—— 一种用于参考语音合成的声学模型，具有增强的风格表达能力。DEX-TTS 基于通用的扩散语音合成框架，包括编码器和适配器，用于处理从参考语音中提取的风格。关键创新包括将风格区分为时不变和时变两个类别，以实现有效的风格提取，以及高泛化能力的编码器和适配器的设计。此外，我们引入了重叠的 patchify 和卷积频率 patch 嵌入策略，以改进基于 DiT 的扩散网络用于 TTS。DEX-TTS 在英语多说话人和情感多说话人数据集上在客观和主观评估方面取得了出色的性能，而无需依赖预训练策略。最后，对于单说话人数据集的一般 TTS 的比较结果验证了我们增强的扩散骨干的有效性。演示请参见此处。