Existing methods for synthesizing 3D human gestures from speech have shown
promising results, but they do not explicitly model the impact of emotions on
the generated gestures. Instead, these methods directly output animations from
speech without control over the expressed emotion. To address this limitation,
we present AMUSE, an emotional speech-driven body animation model based on
latent diffusion. Our observation is that content (i.e., gestures related to
speech rhythm and word utterances), emotion, and personal style are separable.
To account for this, AMUSE maps the driving audio to three disentangled latent
vectors: one for content, one for emotion, and one for personal style. A latent
diffusion model, trained to generate gesture motion sequences, is then
conditioned on these latent vectors. Once trained, AMUSE synthesizes 3D human
gestures directly from speech with control over the expressed emotions and
style by combining the content from the driving speech with the emotion and
style of another speech sequence. Randomly sampling the noise of the diffusion
model further generates variations of the gesture with the same emotional
expressivity. Qualitative, quantitative, and perceptual evaluations demonstrate
that AMUSE outputs realistic gesture sequences. Compared to the state of the
art, the generated gestures are better synchronized with the speech content and
better represent the emotion expressed by the input speech. Our project website
is amuse.is.tue.mpg.de.

AMUSE 是一种基于潜在扩散的情感驱动人体动画模型，通过将语音驱动的内容与另一个语音序列的情绪和风格相结合，从而直接从语音合成具有表达情绪和风格控制的 3D 人体动作序列。