This paper addresses the problem of generating lifelike holistic co-speech
motions for 3D avatars, focusing on two key aspects: variability and
coordination. Variability allows the avatar to exhibit a wide range of motions
even with similar speech content, while coordination ensures a harmonious
alignment among facial expressions, hand gestures, and body poses. We aim to
achieve both with ProbTalk, a unified probabilistic framework designed to
jointly model facial, hand, and body movements in speech. ProbTalk builds on
the variational autoencoder (VAE) architecture and incorporates three core
designs. First, we introduce product quantization (PQ) to the VAE, which
enriches the representation of complex holistic motion. Second, we devise a
novel non-autoregressive model that embeds 2D positional encoding into the
product-quantized representation, thereby preserving essential structure
information of the PQ codes. Last, we employ a secondary stage to refine the
preliminary prediction, further sharpening the high-frequency details. Coupling
these three designs enables ProbTalk to generate natural and diverse holistic
co-speech motions, outperforming several state-of-the-art methods in
qualitative and quantitative evaluations, particularly in terms of realism. Our
code and model will be released for research purposes at
this https URL

通过 ProbTalk 的统一概率框架，以综合协调性和可变性为核心，生成自然多样的整体语音运动，从而比现有方法在质量和数量上都实现了更高的真实度。