Generating speech-consistent body and gesture movements is a long-standing
problem in virtual avatar creation. Previous studies often synthesize pose
movement in a holistic manner, where poses of all joints are generated
simultaneously. Such a straightforward pipeline fails to generate fine-grained
co-speech gestures. One observation is that the hierarchical semantics in
speech and the hierarchical structures of human gestures can be naturally
described into multiple granularities and associated together. To fully utilize
the rich connections between speech audio and human gestures, we propose a
novel framework named Hierarchical Audio-to-Gesture (HA2G) for co-speech
gesture generation. In HA2G, a Hierarchical Audio Learner extracts audio
representations across semantic granularities. A Hierarchical Pose Inferer
subsequently renders the entire human pose gradually in a hierarchical manner.
To enhance the quality of synthesized gestures, we develop a contrastive
learning strategy based on audio-text alignment for better audio
representations. Extensive experiments and human evaluation demonstrate that
the proposed method renders realistic co-speech gestures and outperforms
previous methods in a clear margin. Project page:
this https URL

本文提出了一种名为 HA2G 的新框架，用于语音一致的身体和手势动作的生成。HA2G 使用分层音频学习器和分层姿势推理器提取并逐层渲染全身姿势，同时基于音频文本对齐的对比学习策略提高音频表示的质量，结果表明该方法生成的共语手势更逼真，比以往方法有更高的性能表现。