Animating virtual avatars to make co-speech gestures facilitates various
applications in human-machine interaction. The existing methods mainly rely on
generative adversarial networks (GANs), which typically suffer from notorious
mode collapse and unstable training, thus making it difficult to learn accurate
audio-gesture joint distributions. In this work, we propose a novel
diffusion-based framework, named Diffusion Co-Speech Gesture (DiffGesture), to
effectively capture the cross-modal audio-to-gesture associations and preserve
temporal coherence for high-fidelity audio-driven co-speech gesture generation.
Specifically, we first establish the diffusion-conditional generation process
on clips of skeleton sequences and audio to enable the whole framework. Then, a
novel Diffusion Audio-Gesture Transformer is devised to better attend to the
information from multiple modalities and model the long-term temporal
dependency. Moreover, to eliminate temporal inconsistency, we propose an
effective Diffusion Gesture Stabilizer with an annealed noise sampling
strategy. Benefiting from the architectural advantages of diffusion models, we
further incorporate implicit classifier-free guidance to trade off between
diversity and gesture quality. Extensive experiments demonstrate that
DiffGesture achieves state-of-theart performance, which renders coherent
gestures with better mode coverage and stronger audio correlations. Code is
available at this https URL

本研究提出了 Diffusion Co-Speech Gesture（DiffGesture）框架，该框架可有效捕捉跨模态的音频到手势关联并保持时间上的一致性， 通过弥散模型的设计思路， DiffGesture 可以在质量和多样性之间进行平衡，实现了高保真的音频驱动共话手势生成。