Synthesizing natural interactions between virtual humans and their 3D
environments is critical for numerous applications, such as computer games and
AR/VR experiences. Our goal is to synthesize humans interacting with a given 3D
scene controlled by high-level semantic specifications as pairs of action
categories and object instances, e.g., "sit on the chair". The key challenge of
incorporating interaction semantics into the generation framework is to learn a
joint representation that effectively captures heterogeneous information,
including human body articulation, 3D object geometry, and the intent of the
interaction. To address this challenge, we design a novel transformer-based
generative model, in which the articulated 3D human body surface points and 3D
objects are jointly encoded in a unified latent space, and the semantics of the
interaction between the human and objects are embedded via positional encoding.
Furthermore, inspired by the compositional nature of interactions that humans
can simultaneously interact with multiple objects, we define interaction
semantics as the composition of varying numbers of atomic action-object pairs.
Our proposed generative model can naturally incorporate varying numbers of
atomic interactions, which enables synthesizing compositional human-scene
interactions without requiring composite interaction data. We extend the PROX
dataset with interaction semantic labels and scene instance segmentation to
evaluate our method and demonstrate that our method can generate realistic
human-scene interactions with semantic control. Our perceptual study shows that
our synthesized virtual humans can naturally interact with 3D scenes,
considerably outperforming existing methods. We name our method COINS, for
COmpositional INteraction Synthesis with Semantic Control. Code and data are
available at this https URL

本研究提出了一种名为 COINS 的生成模型，可以在高级语义规格的控制下合成 3D 场景内与虚拟人之间的自然互动，该模型通过编码统一的潜在空间和位置编码来嵌入互动语义，并可以自然地合成多个对象之间的组合互动，实验结果表明该模型可以合成具有语义控制的真实人 - 场景交互。