The problem of audio-to-audio (A2A) style transfer involves replacing the
style features of the source audio with those from the target audio while
preserving the content related attributes of the source audio. In this paper,
we propose an efficient approach, termed as Zero-shot Emotion Style Transfer
(ZEST), that allows the transfer of emotional content present in the given
source audio with the one embedded in the target audio while retaining the
speaker and speech content from the source. The proposed system builds upon
decomposing speech into semantic tokens, speaker representations and emotion
embeddings. Using these factors, we propose a framework to reconstruct the
pitch contour of the given speech signal and train a decoder that reconstructs
the speech signal. The model is trained using a self-supervision based
reconstruction loss. During conversion, the emotion embedding is alone derived
from the target audio, while rest of the factors are derived from the source
audio. In our experiments, we show that, even without using parallel training
data or labels from the source or target audio, we illustrate zero shot emotion
transfer capabilities of the proposed ZEST model using objective and subjective
quality evaluations.

这篇文章介绍了一种高效的零射击情感风格转换的方法，通过分解语音为语义令牌、说话者表示和情感嵌入，并训练一个解码器来重构语音信号，实现将源语音中的情感内容与目标语音嵌入相结合，同时保留源语音中的说话者和语音内容。

零样本音频情感转换与说话人解缠

Zero Shot Audio to Audio Emotion Transfer With Speaker Disentanglement

Despite the recent achievements made in the multi-modal emotion recognition
task, two problems still exist and have not been well investigated: 1) the
relationship between different emotion categories are not utilized, which leads
to sub-optimal performance; and 2) current models fail to cope well with
low-resource emotions, especially for unseen emotions. In this paper, we
propose a modality-transferable model with emotion embeddings to tackle the
aforementioned issues. We use pre-trained word embeddings to represent emotion
categories for textual data. Then, two mapping functions are learned to
transfer these embeddings into visual and acoustic spaces. For each modality,
the model calculates the representation distance between the input sequence and
target emotions and makes predictions based on the distances. By doing so, our
model can directly adapt to the unseen emotions in any modality since we have
their pre-trained embeddings and modality mapping functions. Experiments show
that our model achieves state-of-the-art performance on most of the emotion
categories. In addition, our model also outperforms existing baselines in the
zero-shot and few-shot scenarios for unseen emotions.

本研究提出了一种基于情感嵌入的模态可转移模型，通过学习映射函数将情感嵌入从文本空间转移到视觉和声学空间，进行多模态情感识别，在零 - shot 和 few-shot 情感识别场景下达到了最先进的性能。