Learning to generate diverse scene-aware and goal-oriented human motions in
3D scenes remains challenging due to the mediocre characteristics of the
existing datasets on Human-Scene Interaction (HSI); they only have limited
scale/quality and lack semantics. To fill in the gap, we propose a large-scale
and semantic-rich synthetic HSI dataset, denoted as HUMANISE, by aligning the
captured human motion sequences with various 3D indoor scenes. We automatically
annotate the aligned motions with language descriptions that depict the action
and the unique interacting objects in the scene; e.g., sit on the armchair near
the desk. HUMANISE thus enables a new generation task, language-conditioned
human motion generation in 3D scenes. The proposed task is challenging as it
requires joint modeling of the 3D scene, human motion, and natural language. To
tackle this task, we present a novel scene-and-language conditioned generative
model that can produce 3D human motions of the desirable action interacting
with the specified objects. Our experiments demonstrate that our model
generates diverse and semantically consistent human motions in 3D scenes.

提出了一个名为 HUMANISE 的大规模、语义丰富的合成 HSI 数据集，并开展了一项名为 Language-conditioned Human Motion Generation 的新一代任务，以及相应的在场景和语言指导下的生成模型，该模型能够在 3D 场景中产生多样化且语义一致的人体动作。