Humans constantly interact with daily objects to accomplish tasks. To
understand such interactions, computers need to reconstruct these from cameras
observing whole-body interaction with scenes. This is challenging due to
occlusion between the body and objects, motion blur, depth/scale ambiguities,
and the low image resolution of hands and graspable object parts. To make the
problem tractable, the community focuses either on interacting hands, ignoring
the body, or on interacting bodies, ignoring hands. The GRAB dataset addresses
dexterous whole-body interaction but uses marker-based MoCap and lacks images,
while BEHAVE captures video of body object interaction but lacks hand detail.
We address the limitations of prior work with InterCap, a novel method that
reconstructs interacting whole-bodies and objects from multi-view RGB-D data,
using the parametric whole-body model SMPL-X and known object meshes. To tackle
the above challenges, InterCap uses two key observations: (i) Contact between
the hand and object can be used to improve the pose estimation of both. (ii)
Azure Kinect sensors allow us to set up a simple multi-view RGB-D capture
system that minimizes the effect of occlusion while providing reasonable
inter-camera synchronization. With this method we capture the InterCap dataset,
which contains 10 subjects (5 males and 5 females) interacting with 10 objects
of various sizes and affordances, including contact with the hands or feet. In
total, InterCap has 223 RGB-D videos, resulting in 67,357 multi-view frames,
each containing 6 RGB-D images. Our method provides pseudo ground-truth body
meshes and objects for each video frame. Our InterCap method and dataset fill
an important gap in the literature and support many research directions. Our
data and code are areavailable for research purposes.

本研究提出了一种名为 InterCap 的新方法，利用多视角 RGB-D 数据和参数化的全身模型 SMPL-X 来重建整体身体和对象的互动情况，并构建了一个名为 InterCap 的数据集，填补了现有文献的空白，并支持许多研究方向。