In this paper, we introduce a novel path to $\textit{general}$ human motion
generation by focusing on 2D space. Traditional methods have primarily
generated human motions in 3D, which, while detailed and realistic, are often
limited by the scope of available 3D motion data in terms of both the size and
the diversity. To address these limitations, we exploit extensive availability
of 2D motion data. We present $\textbf{Holistic-Motion2D}$, the first
comprehensive and large-scale benchmark for 2D whole-body motion generation,
which includes over 1M in-the-wild motion sequences, each paired with
high-quality whole-body/partial pose annotations and textual descriptions.
Notably, Holistic-Motion2D is ten times larger than the previously largest 3D
motion dataset. We also introduce a baseline method, featuring innovative
$\textit{whole-body part-aware attention}$ and $\textit{confidence-aware
modeling}$ techniques, tailored for 2D $\underline{\text
T}$ext-driv$\underline{\text{EN}}$ whole-bo$\underline{\text D}$y motion
gen$\underline{\text{ER}}$ation, namely $\textbf{Tender}$. Extensive
experiments demonstrate the effectiveness of $\textbf{Holistic-Motion2D}$ and
$\textbf{Tender}$ in generating expressive, diverse, and realistic human
motions. We also highlight the utility of 2D motion for various downstream
applications and its potential for lifting to 3D motion. The page link is:
this https URL

通过关注二维空间，本研究引入一种新的一般人类动作生成方法，使用名为 Holistic-Motion2D 的大规模二维整体动作数据库和名为 Tender 的创新方法进行生成，展示了二维动作数据在生成多样、真实的人类动作方面的有效性及其对各种下游应用的潜力。

全身动作生成 2D 空间中的可扩展性

Holistic-Motion2D: Scalable Whole-body Human Motion Generation in 2D  Space

This paper addresses the problem of generating whole-body motion from speech.
Despite great successes, prior methods still struggle to produce reasonable and
diverse whole-body motions from speech. This is due to their reliance on
suboptimal representations and a lack of strategies for generating diverse
results. To address these challenges, we present a novel hybrid point
representation to achieve accurate and continuous motion generation, e.g.,
avoiding foot skating, and this representation can be transformed into an
easy-to-use representation, i.e., SMPL-X body mesh, for many applications. To
generate whole-body motion from speech, for facial motion, closely tied to the
audio signal, we introduce an encoder-decoder architecture to achieve
deterministic outcomes. However, for the body and hands, which have weaker
connections to the audio signal, we aim to generate diverse yet reasonable
motions. To boost diversity in motion generation, we propose a contrastive
motion learning method to encourage the model to produce more distinctive
representations. Specifically, we design a robust VQ-VAE to learn a quantized
motion codebook using our hybrid representation. Then, we regress the motion
representation from the audio signal by a translation model employing our
contrastive motion learning method. Experimental results validate the superior
performance and the correctness of our model. The project page is available for
research purposes at this http URL

通过使用混合点表示，并结合对比运动学习方法，本研究提出了一种从语音中生成全身动作的模型，以解决现有方法在从语音中生成多样且合理的全身动作时所面临的挑战。

SpeechAct: 从语音生成全身动作

SpeechAct: Towards Generating Whole-body Motion from Speech

This work targets a novel text-driven whole-body motion generation task,
which takes a given textual description as input and aims at generating
high-quality, diverse, and coherent facial expressions, hand gestures, and body
motions simultaneously. Previous works on text-driven motion generation tasks
mainly have two limitations: they ignore the key role of fine-grained hand and
face controlling in vivid whole-body motion generation, and lack a good
alignment between text and motion. To address such limitations, we propose a
Text-aligned whOle-body Motion generATiOn framework, named HumanTOMATO, which
is the first attempt to our knowledge towards applicable holistic motion
generation in this research area. To tackle this challenging task, our solution
includes two key designs: (1) a Holistic Hierarchical VQ-VAE (aka H$^2$VQ) and
a Hierarchical-GPT for fine-grained body and hand motion reconstruction and
generation with two structured codebooks; and (2) a pre-trained
text-motion-alignment model to help generated motion align with the input
textual description explicitly. Comprehensive experiments verify that our model
has significant advantages in both the quality of generated motions and their
alignment with text.

该研究提出了一个全新的基于文本驱动的整体运动生成任务，以给定的文本描述为输入，并旨在同时生成高质量、多样且连贯的面部表情、手势和身体动作。该研究通过一个名为 HumanTOMATO 的文本对齐全身运动生成框架，解决了以往关于文本驱动动作生成任务的两个限制，即忽视了精细的手部和面部控制在栩栩如生的整体运动生成中的重要作用，并且缺乏文本和运动之间的良好对齐。通过全面实验证明，该模型在生成的动作质量和与文本的对齐方面具有显著优势。