This paper proposes an Expressive Speech Synthesis model that utilizes
token-level latent prosodic variables in order to capture and control
utterance-level attributes, such as character acting voice and speaking style.
Current works aim to explicitly factorize such fine-grained and utterance-level
speech attributes into different representations extracted by modules that
operate in the corresponding level. We show that the fine-grained latent space
also captures coarse-grained information, which is more evident as the
dimension of latent space increases in order to capture diverse prosodic
representations. Therefore, a trade-off arises between the diversity of the
token-level and utterance-level representations and their disentanglement. We
alleviate this issue by first capturing rich speech attributes into a
token-level latent space and then, separately train a prior network that given
the input text, learns utterance-level representations in order to predict the
phoneme-level, posterior latents extracted during the previous step. Both
qualitative and quantitative evaluations are used to demonstrate the
effectiveness of the proposed approach. Audio samples are available in our demo
page.

本论文提出了一种表达性语音合成模型，该模型利用标记级别的潜在韵律变量来捕捉和控制话语级别属性，如角色配音和说话风格，其中的潜在细节级别空间同时也捕捉更粗粒度的信息。

通过预测基于标记的声学潜变量学习话语级表示以用于情感语音合成

Learning utterance-level representations through token-level acoustic latents prediction for Expressive Speech Synthesis

We propose the SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level
Cross-Lingual Speech Representation learning framework. Unlike previous works
on speech representation learning, which learns multilingual contextual speech
embedding at the resolution of an acoustic frame (10-20ms), this work focuses
on learning multimodal (speech-text) multilingual speech embedding at the
resolution of a sentence (5-10s) such that the embedding vector space is
semantically aligned across different languages. We combine state-of-the-art
multilingual acoustic frame-level speech representation learning model XLS-R
with the Language Agnostic BERT Sentence Embedding (LaBSE) model to create an
utterance-level multimodal multilingual speech encoder SAMU-XLSR. Although we
train SAMU-XLSR with only multilingual transcribed speech data, cross-lingual
speech-text and speech-speech associations emerge in its learned representation
space. To substantiate our claims, we use SAMU-XLSR speech encoder in
combination with a pre-trained LaBSE text sentence encoder for cross-lingual
speech-to-text translation retrieval, and SAMU-XLSR alone for cross-lingual
speech-to-speech translation retrieval. We highlight these applications by
performing several cross-lingual text and speech translation retrieval tasks
across several datasets.

提出了 SAMU-XLSR 模型，基于语义对齐的多模态跨语言话语级别语音表示学习框架。通过将 XLS-R 模型和 LaBSE 模型组合，SAMU-XLSR 模型可以提取高质量的多模态多语言话语语音特征，并可以实现跨语音和跨文字之间的相互转换。