Deriving co-speech 3D gestures has seen tremendous progress in virtual avatar
animation. Yet, the existing methods often produce stiff and unreasonable
gestures with unseen human speech inputs due to the limited 3D speech-gesture
data. In this paper, we propose CoCoGesture, a novel framework enabling vivid
and diverse gesture synthesis from unseen human speech prompts. Our key insight
is built upon the custom-designed pretrain-fintune training paradigm. At the
pretraining stage, we aim to formulate a large generalizable gesture diffusion
model by learning the abundant postures manifold. Therefore, to alleviate the
scarcity of 3D data, we first construct a large-scale co-speech 3D gesture
dataset containing more than 40M meshed posture instances across 4.3K speakers,
dubbed GES-X. Then, we scale up the large unconditional diffusion model to 1B
parameters and pre-train it to be our gesture experts. At the finetune stage,
we present the audio ControlNet that incorporates the human voice as condition
prompts to guide the gesture generation. Here, we construct the audio
ControlNet through a trainable copy of our pre-trained diffusion model.
Moreover, we design a novel Mixture-of-Gesture-Experts (MoGE) block to
adaptively fuse the audio embedding from the human speech and the gesture
features from the pre-trained gesture experts with a routing mechanism. Such an
effective manner ensures audio embedding is temporal coordinated with motion
features while preserving the vivid and diverse gesture generation. Extensive
experiments demonstrate that our proposed CoCoGesture outperforms the
state-of-the-art methods on the zero-shot speech-to-gesture generation. The
dataset will be publicly available at: this https URL

我们提出了 CoCoGesture，这是一个能够从未见过的人类语音提示中实现生动多样的手势合成的新框架，通过大规模数据集和预训练模型的优化，在零样本语音转手势生成上表现优于现有方法。