The connection between our 3D surroundings and the descriptive language that
characterizes them would be well-suited for localizing and generating human
motion in context but for one problem. The complexity introduced by multiple
modalities makes capturing this connection challenging with a fixed set of
descriptors. Specifically, closed vocabulary scene encoders, which require
learning text-scene associations from scratch, have been favored in the
literature, often resulting in inaccurate motion grounding. In this paper, we
propose a method that integrates an open vocabulary scene encoder into the
architecture, establishing a robust connection between text and scene. Our
two-step approach starts with pretraining the scene encoder through knowledge
distillation from an existing open vocabulary semantic image segmentation
model, ensuring a shared text-scene feature space. Subsequently, the scene
encoder is fine-tuned for conditional motion generation, incorporating two
novel regularization losses that regress the category and size of the goal
object. Our methodology achieves up to a 30% reduction in the goal object
distance metric compared to the prior state-of-the-art baseline model on the
HUMANISE dataset. This improvement is demonstrated through evaluations
conducted using three implementations of our framework and a perceptual study.
Additionally, our method is designed to seamlessly accommodate future 2D
segmentation methods that provide per-pixel text-aligned features for
distillation.

本文提出了一种方法，将开放词汇场景编码器与架构相结合，建立了文本和场景之间的强大连接；方法通过知识蒸馏从现有的开放词汇语义图像分割模型预训练场景编码器，确保了一个共享的文本 - 场景特征空间，并通过引入两种新的正则化损失，用于回归目标对象的类别和尺寸，在条件运动生成时对场景编码器进行微调；通过在 HUMANISE 数据集上进行的评估和知觉研究，我们的方法相比先前最先进的基准模型，可以使目标对象距离指标减少高达 30%；此外，我们的方法还能无缝适应未来提供每个像素文本对齐特征的 2D 分割方法。

基于开放词汇的情景和文本背景下的基础人体运动生成（GHOST）

GHOST: Grounded Human Motion Generation with Open Vocabulary  Scene-and-Text Contexts

Language-guided scene-aware human motion generation has great significance
for entertainment and robotics. In response to the limitations of existing
datasets, we introduce LaserHuman, a pioneering dataset engineered to
revolutionize Scene-Text-to-Motion research. LaserHuman stands out with its
inclusion of genuine human motions within 3D environments, unbounded free-form
natural language descriptions, a blend of indoor and outdoor scenarios, and
dynamic, ever-changing scenes. Diverse modalities of capture data and rich
annotations present great opportunities for the research of conditional motion
generation, and can also facilitate the development of real-life applications.
Moreover, to generate semantically consistent and physically plausible human
motions, we propose a multi-conditional diffusion model, which is simple but
effective, achieving state-of-the-art performance on existing datasets.

为了解决现有数据集的限制，我们引入了 LaserHuman，这是一个开创性的数据集，旨在革新 Scene-Text-to-Motion 研究。LaserHuman 通过包含 3D 环境中的真实人类动作，自由形式的自然语言描述，室内外场景的混合以及动态、不断变化的场景而脱颖而出。多样的采集数据和丰富的注释为条件运动生成研究提供了巨大机遇，也可以促进实际应用的开发。此外，为了生成语义一致且物理可行的人体动作，我们提出了一个简单但有效的多条件扩散模型，在现有数据集上达到了最先进的性能。