This paper introduces the first text-guided work for generating the sequence
of hand-object interaction in 3D. The main challenge arises from the lack of
labeled data where existing ground-truth datasets are nowhere near
generalizable in interaction type and object category, which inhibits the
modeling of diverse 3D hand-object interaction with the correct physical
implication (e.g., contacts and semantics) from text prompts. To address this
challenge, we propose to decompose the interaction generation task into two
subtasks: hand-object contact generation; and hand-object motion generation.
For contact generation, a VAE-based network takes as input a text and an object
mesh, and generates the probability of contacts between the surfaces of hands
and the object during the interaction. The network learns a variety of local
geometry structure of diverse objects that is independent of the objects'
category, and thus, it is applicable to general objects. For motion generation,
a Transformer-based diffusion model utilizes this 3D contact map as a strong
prior for generating physically plausible hand-object motion as a function of
text prompts by learning from the augmented labeled dataset; where we annotate
text labels from many existing 3D hand and object motion data. Finally, we
further introduce a hand refiner module that minimizes the distance between the
object surface and hand joints to improve the temporal stability of the
object-hand contacts and to suppress the penetration artifacts. In the
experiments, we demonstrate that our method can generate more realistic and
diverse interactions compared to other baseline methods. We also show that our
method is applicable to unseen objects. We will release our model and newly
labeled data as a strong foundation for future research. Codes and data are
available in: this https URL

该研究论文介绍了一项基于文本指导的生成 3D 手物交互序列的工作，并提出了两个子任务：手物接触生成和手物运动生成。通过使用变分自编码器和 Transformer 扩散模型，在文本提示下生成物理上合理的手物交互动作，并通过一种手优化模块改善了接触的时序稳定性和穿透问题。对比基准方法，该方法生成的交互更加真实和多样，并且适用于未知物体。