Co-speech gestures, if presented in the lively form of videos, can achieve
superior visual effects in human-machine interaction. While previous works
mostly generate structural human skeletons, resulting in the omission of
appearance information, we focus on the direct generation of audio-driven
co-speech gesture videos in this work. There are two main challenges: 1) A
suitable motion feature is needed to describe complex human movements with
crucial appearance information. 2) Gestures and speech exhibit inherent
dependencies and should be temporally aligned even of arbitrary length. To
solve these problems, we present a novel motion-decoupled framework to generate
co-speech gesture videos. Specifically, we first introduce a well-designed
nonlinear TPS transformation to obtain latent motion features preserving
essential appearance information. Then a transformer-based diffusion model is
proposed to learn the temporal correlation between gestures and speech, and
performs generation in the latent motion space, followed by an optimal motion
selection module to produce long-term coherent and consistent gesture videos.
For better visual perception, we further design a refinement network focusing
on missing details of certain areas. Extensive experimental results show that
our proposed framework significantly outperforms existing approaches in both
motion and video-related evaluations. Our code, demos, and more resources are
available at this https URL

本文提出了一种新的运动解耦框架来生成音频驱动的共语手势视频，通过引入非线性 TPS 变换和基于变压器的扩散模型，学习手势和语音之间的时序相关性，并利用优化运动选择模块生成长期连贯和一致的手势视频，最后设计了一个细节修复网络来增强视觉感知，实验结果表明我们的框架在动作和视频评估方面明显优于现有方法。