In this paper, we propose a novel cascaded diffusion-based generative
framework for text-driven human motion synthesis, which exploits a strategy
named GradUally Enriching SyntheSis (GUESS as its abbreviation). The strategy
sets up generation objectives by grouping body joints of detailed skeletons in
close semantic proximity together and then replacing each of such joint group
with a single body-part node. Such an operation recursively abstracts a human
pose to coarser and coarser skeletons at multiple granularity levels. With
gradually increasing the abstraction level, human motion becomes more and more
concise and stable, significantly benefiting the cross-modal motion synthesis
task. The whole text-driven human motion synthesis problem is then divided into
multiple abstraction levels and solved with a multi-stage generation framework
with a cascaded latent diffusion model: an initial generator first generates
the coarsest human motion guess from a given text description; then, a series
of successive generators gradually enrich the motion details based on the
textual description and the previous synthesized results. Notably, we further
integrate GUESS with the proposed dynamic multi-condition fusion mechanism to
dynamically balance the cooperative effects of the given textual condition and
synthesized coarse motion prompt in different generation stages. Extensive
experiments on large-scale datasets verify that GUESS outperforms existing
state-of-the-art methods by large margins in terms of accuracy, realisticness,
and diversity. Code is available at this https URL

本研究提出了一种基于级联扩散的生成框架，用于文本驱动的人体动作合成，该框架利用了一种名为 GradUally Enriching SyntheSis（GUESS）的策略。通过将语义上紧密相连的身体关节进行聚类分组，并用单个身体部分节点替换每个关节组，该策略逐步将人体姿态抽象为多个粒度级别上的更粗糙的骨架。采用逐渐增加的抽象级别，人体动作变得更加简洁和稳定，显著改善了跨模态动作合成任务。然后，将整个文本驱动的人体动作合成问题划分为多个抽象级别，并利用级联潜在扩散模型的多阶段生成框架解决：初始生成器首先从给定的文本描述中生成最粗糙的人体动作猜测；然后，一系列连续的生成器根据文本描述和之前合成的结果逐渐丰富动作细节。值得注意的是，我们进一步将 GUESS 与所提出的动态多条件融合机制相结合，以动态平衡给定的文本条件和合成粗动作提示在不同生成阶段的合作效应。大规模数据集上的广泛实验证明，GUESS 在准确性、逼真度和多样性方面优于现有的最先进方法。

文本驱动的人体动作生成的逐渐丰富合成

GUESS:GradUally Enriching SyntheSis for Text-Driven Human Motion  Generation

Our paper aims to generate diverse and realistic animal motion sequences from
textual descriptions, without a large-scale animal text-motion dataset. While
the task of text-driven human motion synthesis is already extensively studied
and benchmarked, it remains challenging to transfer this success to other
skeleton structures with limited data. In this work, we design a model
architecture that imitates Generative Pretraining Transformer (GPT), utilizing
prior knowledge learned from human data to the animal domain. We jointly train
motion autoencoders for both animal and human motions and at the same time
optimize through the similarity scores among human motion encoding, animal
motion encoding, and text CLIP embedding. Presenting the first solution to this
problem, we are able to generate animal motions with high diversity and
fidelity, quantitatively and qualitatively outperforming the results of
training human motion generation baselines on animal data. Additionally, we
introduce AnimalML3D, the first text-animal motion dataset with 1240 animation
sequences spanning 36 different animal identities. We hope this dataset would
mediate the data scarcity problem in text-driven animal motion generation,
providing a new playground for the research community.

本文旨在从文本描述中生成多样且逼真的动物运动序列，无需大规模动物文本 - 运动数据集。通过设计一个模仿 GPT 的模型架构，利用从人类数据中学到的先验知识应用于动物领域，我们可以联合训练动物和人类运动的运动自编码器，并同时优化人类运动编码、动物运动编码和文本 CLIP 嵌入之间的相似性分数。我们创造性地解决了该问题，并能够生成具有高多样性和逼真度的动物运动，数量和质量上均优于在动物数据上训练人类运动生成基线的结果。此外，我们还推出了 AnimalML3D，第一个包含 1240 个动画序列和 36 种不同动物身份的文本 - 动物运动数据集。我们希望这个数据集能够解决文本驱动动物运动生成中的数据稀缺问题，并为研究社区提供一个新的研究平台。