This paper addresses the problem of generating whole-body motion from speech.
Despite great successes, prior methods still struggle to produce reasonable and
diverse whole-body motions from speech. This is due to their reliance on
suboptimal representations and a lack of strategies for generating diverse
results. To address these challenges, we present a novel hybrid point
representation to achieve accurate and continuous motion generation, e.g.,
avoiding foot skating, and this representation can be transformed into an
easy-to-use representation, i.e., SMPL-X body mesh, for many applications. To
generate whole-body motion from speech, for facial motion, closely tied to the
audio signal, we introduce an encoder-decoder architecture to achieve
deterministic outcomes. However, for the body and hands, which have weaker
connections to the audio signal, we aim to generate diverse yet reasonable
motions. To boost diversity in motion generation, we propose a contrastive
motion learning method to encourage the model to produce more distinctive
representations. Specifically, we design a robust VQ-VAE to learn a quantized
motion codebook using our hybrid representation. Then, we regress the motion
representation from the audio signal by a translation model employing our
contrastive motion learning method. Experimental results validate the superior
performance and the correctness of our model. The project page is available for
research purposes at this http URL

通过使用混合点表示，并结合对比运动学习方法，本研究提出了一种从语音中生成全身动作的模型，以解决现有方法在从语音中生成多样且合理的全身动作时所面临的挑战。