Textless speech-to-speech translation systems are rapidly advancing, thanks
to the integration of self-supervised learning techniques. However, existing
state-of-the-art systems fall short when it comes to capturing and transferring
expressivity accurately across different languages. Expressivity plays a vital
role in conveying emotions, nuances, and cultural subtleties, thereby enhancing
communication across diverse languages. To address this issue this study
presents a novel method that operates at the discrete speech unit level and
leverages multilingual emotion embeddings to capture language-agnostic
information. Specifically, we demonstrate how these embeddings can be used to
effectively predict the pitch and duration of speech units in the target
language. Through objective and subjective experiments conducted on a
French-to-English translation task, our findings highlight the superior
expressivity transfer achieved by our approach compared to current
state-of-the-art systems.

通过离散语音单元级别的多语情感嵌入，以及有效地预测目标语言中的语音单元的音高和持续时间，本研究展示了一种能够有效捕捉跨不同语言的表达能力的新方法，通过客观和主观实验，结果显示本方法在提高传达情感和文化细微差别方面优于当前先进系统的表达能力转移。