Human-object contact serves as a strong cue to understand how humans
physically interact with objects. Nevertheless, it is not widely explored to
utilize human-object contact information for the joint reconstruction of 3D
human and object from a single image. In this work, we present a novel joint 3D
human-object reconstruction method (CONTHO) that effectively exploits contact
information between humans and objects. There are two core designs in our
system: 1) 3D-guided contact estimation and 2) contact-based 3D human and
object refinement. First, for accurate human-object contact estimation, CONTHO
initially reconstructs 3D humans and objects and utilizes them as explicit 3D
guidance for contact estimation. Second, to refine the initial reconstructions
of 3D human and object, we propose a novel contact-based refinement Transformer
that effectively aggregates human features and object features based on the
estimated human-object contact. The proposed contact-based refinement prevents
the learning of erroneous correlation between human and object, which enables
accurate 3D reconstruction. As a result, our CONTHO achieves state-of-the-art
performance in both human-object contact estimation and joint reconstruction of
3D human and object. The code is publicly available at
this https URL

该研究提出了一种利用人 - 物接触信息进行三维人体与物体联合重建的新方法 (CONTHO)，通过 3D 引导接触估计和基于接触的三维人体与物体细化，实现了更准确的重建，取得了最先进的性能。

基于接触优化的转换器进行 3D 人体和物体联合重建

Joint Reconstruction of 3D Human and Object via Contact-Based Refinement  Transformer

Humans constantly contact objects to move and perform tasks. Thus, detecting
human-object contact is important for building human-centered artificial
intelligence. However, there exists no robust method to detect contact between
the body and the scene from an image, and there exists no dataset to learn such
a detector. We fill this gap with HOT ("Human-Object conTact"), a new dataset
of human-object contacts for images. To build HOT, we use two data sources: (1)
We use the PROX dataset of 3D human meshes moving in 3D scenes, and
automatically annotate 2D image areas for contact via 3D mesh proximity and
projection. (2) We use the V-COCO, HAKE and Watch-n-Patch datasets, and ask
trained annotators to draw polygons for the 2D image areas where contact takes
place. We also annotate the involved body part of the human body. We use our
HOT dataset to train a new contact detector, which takes a single color image
as input, and outputs 2D contact heatmaps as well as the body-part labels that
are in contact. This is a new and challenging task that extends current
foot-ground or hand-object contact detectors to the full generality of the
whole body. The detector uses a part-attention branch to guide contact
estimation through the context of the surrounding body parts and scene. We
evaluate our detector extensively, and quantitative results show that our model
outperforms baselines, and that all components contribute to better
performance. Results on images from an online repository show reasonable
detections and generalizability.

本文介绍了一种用于检测人与周围物体接触的新数据集 HOT，并使用该数据集训练了一个新的接触检测器，它可以输出 2D 接触热图以及处于接触状态的身体部位标签。该检测器采用了部分注意分支来通过周围身体部分和场景的上下文指导接触估计，并且定量的结果显示，该模型的性能优于基线，并且所有组件都对提高性能有贡献。