We introduce a data capture system and a new dataset named HO-Cap that can be
used to study 3D reconstruction and pose tracking of hands and objects in
videos. The capture system uses multiple RGB-D cameras and a HoloLens headset
for data collection, avoiding the use of expensive 3D scanners or mocap
systems. We propose a semi-automatic method to obtain annotations of shape and
pose of hands and objects in the collected videos, which significantly reduces
the required annotation time compared to manual labeling. With this system, we
captured a video dataset of humans using objects to perform different tasks, as
well as simple pick-and-place and handover of an object from one hand to the
other, which can be used as human demonstrations for embodied AI and robot
manipulation research. Our data capture setup and annotation framework can be
used by the community to reconstruct 3D shapes of objects and human hands and
track their poses in videos.

我们引入了一个数据采集系统和一个名为 HO-Cap 的新数据集，可以用于研究视频中手和物体的 3D 重建和姿态跟踪。该系统使用多个 RGB-D 相机和 HoloLens 头戴式显示器进行数据采集，避免使用昂贵的 3D 扫描仪或动作捕捉系统。我们提出了一种半自动的方法来获得手和物体形状和姿态的注释，与手动标注相比，大大减少了所需的标注时间。通过这个系统，我们捕捉了一个视频数据集，其中人类使用物体来执行不同的任务，以及简单的拾取和移交某个物体给另一只手，这可以作为人类示范用于具身人工智能和机器人操纵研究。我们的数据采集设置和注释框架可以被社区用来在视频中重建物体和人手的 3D 形状并跟踪它们的姿势。