Text-driven motion generation has achieved substantial progress with the
emergence of diffusion models. However, existing methods still struggle to
generate complex motion sequences that correspond to fine-grained descriptions,
depicting detailed and accurate spatio-temporal actions. This lack of fine
controllability limits the usage of motion generation to a larger audience. To
tackle these challenges, we present FineMoGen, a diffusion-based motion
generation and editing framework that can synthesize fine-grained motions, with
spatial-temporal composition to the user instructions. Specifically, FineMoGen
builds upon diffusion model with a novel transformer architecture dubbed
Spatio-Temporal Mixture Attention (SAMI). SAMI optimizes the generation of the
global attention template from two perspectives: 1) explicitly modeling the
constraints of spatio-temporal composition; and 2) utilizing sparsely-activated
mixture-of-experts to adaptively extract fine-grained features. To facilitate a
large-scale study on this new fine-grained motion generation task, we
contribute the HuMMan-MoGen dataset, which consists of 2,968 videos and 102,336
fine-grained spatio-temporal descriptions. Extensive experiments validate that
FineMoGen exhibits superior motion generation quality over state-of-the-art
methods. Notably, FineMoGen further enables zero-shot motion editing
capabilities with the aid of modern large language models (LLM), which
faithfully manipulates motion sequences with fine-grained instructions. Project
Page: this https URL

FineMoGen 是一个基于扩散模型的运动生成和编辑框架，可以合成细粒度的运动，并以用户指令为依据进行空间 - 时间组合。

FineMoGen: 细粒度时空运动生成与编辑

FineMoGen: Fine-Grained Spatio-Temporal Motion Generation and Editing

This work targets a novel text-driven whole-body motion generation task,
which takes a given textual description as input and aims at generating
high-quality, diverse, and coherent facial expressions, hand gestures, and body
motions simultaneously. Previous works on text-driven motion generation tasks
mainly have two limitations: they ignore the key role of fine-grained hand and
face controlling in vivid whole-body motion generation, and lack a good
alignment between text and motion. To address such limitations, we propose a
Text-aligned whOle-body Motion generATiOn framework, named HumanTOMATO, which
is the first attempt to our knowledge towards applicable holistic motion
generation in this research area. To tackle this challenging task, our solution
includes two key designs: (1) a Holistic Hierarchical VQ-VAE (aka H$^2$VQ) and
a Hierarchical-GPT for fine-grained body and hand motion reconstruction and
generation with two structured codebooks; and (2) a pre-trained
text-motion-alignment model to help generated motion align with the input
textual description explicitly. Comprehensive experiments verify that our model
has significant advantages in both the quality of generated motions and their
alignment with text.

该研究提出了一个全新的基于文本驱动的整体运动生成任务，以给定的文本描述为输入，并旨在同时生成高质量、多样且连贯的面部表情、手势和身体动作。该研究通过一个名为 HumanTOMATO 的文本对齐全身运动生成框架，解决了以往关于文本驱动动作生成任务的两个限制，即忽视了精细的手部和面部控制在栩栩如生的整体运动生成中的重要作用，并且缺乏文本和运动之间的良好对齐。通过全面实验证明，该模型在生成的动作质量和与文本的对齐方面具有显著优势。

HumanTOMATO: 文本对齐的全身动作生成

HumanTOMATO: Text-aligned Whole-body Motion Generation

Generating 3D human motion based on textual descriptions has been a research
focus in recent years. It requires the generated motion to be diverse, natural,
and conform to the textual description. Due to the complex spatio-temporal
nature of human motion and the difficulty in learning the cross-modal
relationship between text and motion, text-driven motion generation is still a
challenging problem. To address these issues, we propose \textbf{AttT2M}, a
two-stage method with multi-perspective attention mechanism: \textbf{body-part
attention} and \textbf{global-local motion-text attention}. The former focuses
on the motion embedding perspective, which means introducing a body-part
spatio-temporal encoder into VQ-VAE to learn a more expressive discrete latent
space. The latter is from the cross-modal perspective, which is used to learn
the sentence-level and word-level motion-text cross-modal relationship. The
text-driven motion is finally generated with a generative transformer.
Extensive experiments conducted on HumanML3D and KIT-ML demonstrate that our
method outperforms the current state-of-the-art works in terms of qualitative
and quantitative evaluation, and achieve fine-grained synthesis and
action2motion. Our code is in this https URL

基于文本描述生成的 3D 人体动作一直以来都是一个研究焦点，本文提出了使用多角度注意机制的两阶段方法，即基于人体局部和整体的运动注意以及运动和文本跨模态的全局局部注意机制，通过生成变压器实现文本驱动的运动生成，在 HumanML3D 和 KIT-ML 上的实验证明了我们方法在定性和定量评估方面优于现有的技术，并实现了精细合成和动作生成。

基于文本的人体运动生成与多视角关注机制

AttT2M: Text-Driven Human Motion Generation with Multi-Perspective  Attention Mechanism

Human motion modeling is important for many modern graphics applications,
which typically require professional skills. In order to remove the skill
barriers for laymen, recent motion generation methods can directly generate
human motions conditioned on natural languages. However, it remains challenging
to achieve diverse and fine-grained motion generation with various text inputs.
To address this problem, we propose MotionDiffuse, the first diffusion
model-based text-driven motion generation framework, which demonstrates several
desired properties over existing methods. 1) Probabilistic Mapping. Instead of
a deterministic language-motion mapping, MotionDiffuse generates motions
through a series of denoising steps in which variations are injected. 2)
Realistic Synthesis. MotionDiffuse excels at modeling complicated data
distribution and generating vivid motion sequences. 3) Multi-Level
Manipulation. MotionDiffuse responds to fine-grained instructions on body
parts, and arbitrary-length motion synthesis with time-varied text prompts. Our
experiments show MotionDiffuse outperforms existing SoTA methods by convincing
margins on text-driven motion generation and action-conditioned motion
generation. A qualitative analysis further demonstrates MotionDiffuse's
controllability for comprehensive motion generation. Homepage:
this https URL

MotionDiffuse 是一种基于扩散模型的文本驱动运动生成框架，可以通过概率映射、真实的合成方式，以及多级操作等方法，有效地生成复杂和多种变化的人类运动。