People naturally conduct spontaneous body motions to enhance their speeches
while giving talks. Body motion generation from speech is inherently difficult
due to the non-deterministic mapping from speech to body motions. Most existing
works map speech to motion in a deterministic way by conditioning on certain
styles, leading to sub-optimal results. Motivated by studies in linguistics, we
decompose the co-speech motion into two complementary parts: pose modes and
rhythmic dynamics. Accordingly, we introduce a novel freeform motion generation
model (FreeMo) by equipping a two-stream architecture, i.e., a pose mode branch
for primary posture generation, and a rhythmic motion branch for rhythmic
dynamics synthesis. On one hand, diverse pose modes are generated by
conditional sampling in a latent space, guided by speech semantics. On the
other hand, rhythmic dynamics are synced with the speech prosody. Extensive
experiments demonstrate the superior performance against several baselines, in
terms of motion diversity, quality and syncing with speech. Code and
pre-trained models will be publicly available through
this https URL

该论文提出了一种新型的自由形式动作生成模型（FreeMo），通过将共同话语动作分解为姿态模式和节奏动力学，利用两条流架构实现姿态的多样性生成和语音韵律同步的节奏动力学合成，从而在几个基准测试中展示出优秀的性能。