Co-speech gesture is crucial for human-machine interaction and digital
entertainment. While previous works mostly map speech audio to human skeletons
(e.g., 2D keypoints), directly generating speakers' gestures in the image
domain remains unsolved. In this work, we formally define and study this
challenging problem of audio-driven co-speech gesture video generation, i.e.,
using a unified framework to generate speaker image sequence driven by speech
audio. Our key insight is that the co-speech gestures can be decomposed into
common motion patterns and subtle rhythmic dynamics. To this end, we propose a
novel framework, Audio-driveN Gesture vIdeo gEneration (ANGIE), to effectively
capture the reusable co-speech gesture patterns as well as fine-grained
rhythmic movements. To achieve high-fidelity image sequence generation, we
leverage an unsupervised motion representation instead of a structural human
body prior (e.g., 2D skeletons). Specifically, 1) we propose a vector quantized
motion extractor (VQ-Motion Extractor) to summarize common co-speech gesture
patterns from implicit motion representation to codebooks. 2) Moreover, a
co-speech gesture GPT with motion refinement (Co-Speech GPT) is devised to
complement the subtle prosodic motion details. Extensive experiments
demonstrate that our framework renders realistic and vivid co-speech gesture
video. Demo video and more resources can be found in:
this https URL

本研究旨在解决语音驱动的共同语言手势图像序列生成问题，提出了一种名为 ANGIE 的框架，使用向量量化运动提取器和共同语言 GPT，以有效地捕捉可重用的共同语言手势模式和细粒度节奏变化，从而实现高保真度的图像序列生成。