Expressive Text-to-Speech (TTS) using reference speech has been studied
extensively to synthesize natural speech, but there are limitations to
obtaining well-represented styles and improving model generalization ability.
In this study, we present Diffusion-based EXpressive TTS (DEX-TTS), an acoustic
model designed for reference-based speech synthesis with enhanced style
representations. Based on a general diffusion TTS framework, DEX-TTS includes
encoders and adapters to handle styles extracted from reference speech. Key
innovations contain the differentiation of styles into time-invariant and
time-variant categories for effective style extraction, as well as the design
of encoders and adapters with high generalization ability. In addition, we
introduce overlapping patchify and convolution-frequency patch embedding
strategies to improve DiT-based diffusion networks for TTS. DEX-TTS yields
outstanding performance in terms of objective and subjective evaluation in
English multi-speaker and emotional multi-speaker datasets, without relying on
pre-training strategies. Lastly, the comparison results for the general TTS on
a single-speaker dataset verify the effectiveness of our enhanced diffusion
backbone. Demos are available here.

基于参考语音的语音合成已经被广泛研究来合成自然语音，但在获取良好的表达风格和提高模型泛化能力方面存在限制。本研究提出了基于扩散的表达性语音合成 (DEX-TTS)—— 一种用于参考语音合成的声学模型，具有增强的风格表达能力。DEX-TTS 基于通用的扩散语音合成框架，包括编码器和适配器，用于处理从参考语音中提取的风格。关键创新包括将风格区分为时不变和时变两个类别，以实现有效的风格提取，以及高泛化能力的编码器和适配器的设计。此外，我们引入了重叠的 patchify 和卷积频率 patch 嵌入策略，以改进基于 DiT 的扩散网络用于 TTS。DEX-TTS 在英语多说话人和情感多说话人数据集上在客观和主观评估方面取得了出色的性能，而无需依赖预训练策略。最后，对于单说话人数据集的一般 TTS 的比较结果验证了我们增强的扩散骨干的有效性。演示请参见此处。

基于扩散的具有风格建模的表达性文本转语音系统对时间变异性进行研究

DEX-TTS: Diffusion-based EXpressive Text-to-Speech with Style Modeling  on Time Variability

We propose a novel method for generating abstract art. First an autoencoder
is trained to encode and decode the style representations of images, which are
extracted from source images with a pretrained VGG network. Then, the decoder
component of the autoencoder is extracted and used as a generator in a GAN. The
generator works with an ensemble of discriminators. Each discriminator takes
different style representations of the same images, and the generator is
trained to create images that create convincing style representations in order
to deceive all of the generators. The generator is also trained to maximize a
diversity term. The resulting images had a surreal, geometric quality. We call
our approach ARTEMIS (ARTistic Encoder- Multi- Discriminators Including
Self-Attention), as it uses the self-attention layers and an encoder-decoder
architecture.

我们提出了一种新颖的生成抽象艺术的方法，通过训练自动编码器，提取源图像的风格表示，并使用生成对抗网络中的解码器作为生成器，结合多个鉴别器使生成器能够欺骗全部鉴别器，并最大化多样性项，最终生成具有超现实几何特性的图像。我们将其称为 ARTEMIS（艺术编码器 - 包括自注意力的多鉴别器），因为它使用了自注意力层和编码器 - 解码器架构。