Our work aims to obtain 3D reconstruction of hands and manipulated objects
from monocular videos. Reconstructing hand-object manipulations holds a great
potential for robotics and learning from human demonstrations. The supervised
learning approach to this problem, however, requires 3D supervision and remains
limited to constrained laboratory settings and simulators for which 3D ground
truth is available. In this paper we first propose a learning-free fitting
approach for hand-object reconstruction which can seamlessly handle two-hand
object interactions. Our method relies on cues obtained with common methods for
object detection, hand pose estimation and instance segmentation. We
quantitatively evaluate our approach and show that it can be applied to
datasets with varying levels of difficulty for which training data is
unavailable.

本文旨在从单目视频中获取手部和操纵对象的三维重构，提出了一种学习免费的拟合方法以应用于无法获取训练数据且具有不同难度等级的数据集，并量化评估了该方法。

从 RGB 视频中实现无限制联合手部和物体重建

Towards unconstrained joint hand-object reconstruction from RGB videos

We propose a method for annotating images of a hand manipulating an object
with the 3D poses of both the hand and the object, together with a dataset
created using this method. Our motivation is the current lack of annotated real
images for this problem, as estimating the 3D poses is challenging, mostly
because of the mutual occlusions between the hand and the object. To tackle
this challenge, we capture sequences with one or several RGB-D cameras and
jointly optimize the 3D hand and object poses over all the frames
simultaneously. This method allows us to automatically annotate each frame with
accurate estimates of the poses, despite large mutual occlusions. With this
method, we created HO-3D, the first markerless dataset of color images with 3D
annotations for both the hand and object. This dataset is currently made of
77,558 frames, 68 sequences, 10 persons, and 10 objects. Using our dataset, we
develop a single RGB image-based method to predict the hand pose when
interacting with objects under severe occlusions and show it generalizes to
objects not seen in the dataset.

提出了一种用于注释手操作对象图像的 3D 姿态方法，以及使用该方法创建的数据集并开发了一种基于 RGB 图像的手姿态预测方法。