In this study, we tackle the complex task of generating 3D human-object
interactions (HOI) from textual descriptions in a zero-shot text-to-3D manner.
We identify and address two key challenges: the unsatisfactory outcomes of
direct text-to-3D methods in HOI, largely due to the lack of paired
text-interaction data, and the inherent difficulties in simultaneously
generating multiple concepts with complex spatial relationships. To effectively
address these issues, we present InterFusion, a two-stage framework
specifically designed for HOI generation. InterFusion involves human pose
estimations derived from text as geometric priors, which simplifies the
text-to-3D conversion process and introduces additional constraints for
accurate object generation. At the first stage, InterFusion extracts 3D human
poses from a synthesized image dataset depicting a wide range of interactions,
subsequently mapping these poses to interaction descriptions. The second stage
of InterFusion capitalizes on the latest developments in text-to-3D generation,
enabling the production of realistic and high-quality 3D HOI scenes. This is
achieved through a local-global optimization process, where the generation of
human body and object is optimized separately, and jointly refined with a
global optimization of the entire scene, ensuring a seamless and contextually
coherent integration. Our experimental results affirm that InterFusion
significantly outperforms existing state-of-the-art methods in 3D HOI
generation.

本研究利用 InterFusion 框架，通过从文本中提取到的人体姿势估计作为几何先验，实现了具有复杂空间关系的多个概念的高质量三维人物 - 物体交互场景的生成，并且结果表明 InterFusion 在三维 HOI 生成方面显著优于现有的最先进方法。

文本驱动的三维人物物体交互生成

InterFusion: Text-Driven Generation of 3D Human-Object Interaction

We address the problem of generating realistic 3D human-object interactions
(HOIs) driven by textual prompts. Instead of a single model, our key insight is
to take a modular design and decompose the complex task into simpler sub-tasks.
We first develop a dual-branch diffusion model (HOI-DM) to generate both human
and object motions conditioning on the input text, and encourage coherent
motions by a cross-attention communication module between the human and object
motion generation branches. We also develop an affordance prediction diffusion
model (APDM) to predict the contacting area between the human and object during
the interactions driven by the textual prompt. The APDM is independent of the
results by the HOI-DM and thus can correct potential errors by the latter.
Moreover, it stochastically generates the contacting points to diversify the
generated motions. Finally, we incorporate the estimated contacting points into
the classifier-guidance to achieve accurate and close contact between humans
and objects. To train and evaluate our approach, we annotate BEHAVE dataset
with text descriptions. Experimental results demonstrate that our approach is
able to produce realistic HOIs with various interactions and different types of
objects.

我们研究了通过文本指令生成逼真的三维人 - 物互动问题。我们采用模块化设计，将复杂任务分解为简单子任务。我们开发了双分支扩散模型（HOI-DM）来生成人和物体的动作，并通过人和物体动作生成分支之间的交叉注意力通信模块来促进一致的动作。我们还开发了互动预测扩散模型（APDM）来预测通过文本指令驱动的人和物体之间的接触区域。APDM 相对于 HOI-DM 结果是独立的，并且可以纠正后者的潜在错误。此外，它可以随机生成接触点，以多样化生成的动作。最后，我们将估计的接触点合并到分类器引导中，实现准确和密切的人与物之间的接触。通过注释 BEHAVE 数据集的文本描述来训练和评估我们的方法。实验结果表明，我们的方法能够产生具有各种互动和不同类型物体的逼真的人 - 物互动。

HOI-Diff: 使用扩散模型基于文本驱动合成的三维人 - 物交互

HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using  Diffusion Models

This paper addresses a novel task of anticipating 3D human-object
interactions (HOIs). Most existing research on HOI synthesis lacks
comprehensive whole-body interactions with dynamic objects, e.g., often limited
to manipulating small or static objects. Our task is significantly more
challenging, as it requires modeling dynamic objects with various shapes,
capturing whole-body motion, and ensuring physically valid interactions. To
this end, we propose InterDiff, a framework comprising two key steps: (i)
interaction diffusion, where we leverage a diffusion model to encode the
distribution of future human-object interactions; (ii) interaction correction,
where we introduce a physics-informed predictor to correct denoised HOIs in a
diffusion step. Our key insight is to inject prior knowledge that the
interactions under reference with respect to contact points follow a simple
pattern and are easily predictable. Experiments on multiple human-object
interaction datasets demonstrate the effectiveness of our method for this task,
capable of producing realistic, vivid, and remarkably long-term 3D HOI
predictions.

该论文提出了一种预测三维人物 - 物体交互（HOIs）的新方法。通过引入交互扩散和交互校正两个关键步骤，利用扩散模型编码未来人物 - 物体交互的分布，并引入物理感知预测器以纠正去噪后的 HOIs，该方法在多个数据集上的实验证明了其在生成逼真、生动且长期的三维 HOI 预测方面的有效性。