We present TeSMo, a method for text-controlled scene-aware motion generation
based on denoising diffusion models. Previous text-to-motion methods focus on
characters in isolation without considering scenes due to the limited
availability of datasets that include motion, text descriptions, and
interactive scenes. Our approach begins with pre-training a scene-agnostic
text-to-motion diffusion model, emphasizing goal-reaching constraints on
large-scale motion-capture datasets. We then enhance this model with a
scene-aware component, fine-tuned using data augmented with detailed scene
information, including ground plane and object shapes. To facilitate training,
we embed annotated navigation and interaction motions within scenes. The
proposed method produces realistic and diverse human-object interactions, such
as navigation and sitting, in different scenes with various object shapes,
orientations, initial body positions, and poses. Extensive experiments
demonstrate that our approach surpasses prior techniques in terms of the
plausibility of human-scene interactions, as well as the realism and variety of
the generated motions. Code will be released upon publication of this work at
this https URL

基于去噪扩散模型，我们提出了一种用于文本控制的场景感知运动生成方法。该方法通过预训练场景不可知的文本到运动扩散模型，并利用包含详细场景信息的增强数据对模型进行微调，产生逼真多样的人 - 场景互动。

在场景中根据文本控制生成人类互动动作

Generating Human Interaction Motions in Scenes with Text Control

This paper addresses the problem of generating 3D interactive human motion
from text. Given a textual description depicting the actions of different body
parts in contact with objects, we synthesize sequences of 3D body poses that
are visually natural and physically plausible. Yet, this task poses a
significant challenge due to the inadequate consideration of interactions by
physical contacts in both motion and textual descriptions, leading to unnatural
and implausible sequences. To tackle this challenge, we create a novel dataset
named RICH-CAT, representing ``Contact-Aware Texts'' constructed from the RICH
dataset. RICH-CAT comprises high-quality motion, accurate human-object contact
labels, and detailed textual descriptions, encompassing over 8,500 motion-text
pairs across 26 indoor/outdoor actions. Leveraging RICH-CAT, we propose a novel
approach named CATMO for text-driven interactive human motion synthesis that
explicitly integrates human body contacts as evidence. We employ two VQ-VAE
models to encode motion and body contact sequences into distinct yet
complementary latent spaces and an intertwined GPT for generating human motions
and contacts in a mutually conditioned manner. Additionally, we introduce a
pre-trained text encoder to learn textual embeddings that better discriminate
among various contact types, allowing for more precise control over synthesized
motions and contacts. Our experiments demonstrate the superior performance of
our approach compared to existing text-to-motion methods, producing stable,
contact-aware motion sequences. Code and data will be available for research
purposes.

从文本中生成三维交互式人体动作的研究，通过引入人体与物体接触的联系，提出了一种新的方法 CATMO，整合了运动和接触序列，通过学习文本嵌入来生成稳定的动作序列。