3D reconstruction of hand-object manipulations is important for emulating
human actions. Most methods dealing with challenging object manipulation
scenarios, focus on hands reconstruction in isolation, ignoring physical and
kinematic constraints due to object contact. Some approaches produce more
realistic results by jointly reconstructing 3D hand-object interactions.
However, they focus on coarse pose estimation or rely upon known hand and
object shapes. We propose the first approach for realistic 3D hand-object shape
and pose reconstruction from a single depth map. Unlike previous work, our
voxel-based reconstruction network regresses the vertex coordinates of a hand
and an object and reconstructs more realistic interaction. Our pipeline
additionally predicts voxelized hand-object shapes, having a one-to-one mapping
to the input voxelized depth. Thereafter, we exploit the graph nature of the
hand and object shapes, by utilizing the recent GraFormer network with
positional embedding to reconstruct shapes from template meshes. In addition,
we show the impact of adding another GraFormer component that refines the
reconstructed shapes based on the hand-object interactions and its ability to
reconstruct more accurate object shapes. We perform an extensive evaluation on
the HO-3D and DexYCB datasets and show that our method outperforms existing
approaches in hand reconstruction and produces plausible reconstructions for
the objects

利用深度图进行真实的 3D 手物体形状和姿态重建方法，通过基于体素的重建网络回归手和物体的顶点坐标，预测体素化的手物体形状，并利用图形网络通过位置嵌入从模板网格中重建形状，并表明手物体交互对于更准确的物体形状重建有影响，该方法在手重建和物体重建方面优于现有方法，能够产生可信的重建结果。

ShapeGraFormer: 基于 GraFormer 的手 - 物体重建网络

ShapeGraFormer: GraFormer-Based Network for Hand-Object Reconstruction  from a Single Depth Map

Modeling hand-object manipulations is essential for understanding how humans
interact with their environment. While of practical importance, estimating the
pose of hands and objects during interactions is challenging due to the large
mutual occlusions that occur during manipulation. Recent efforts have been
directed towards fully-supervised methods that require large amounts of labeled
training samples. Collecting 3D ground-truth data for hand-object interactions,
however, is costly, tedious, and error-prone. To overcome this challenge we
present a method to leverage photometric consistency across time when
annotations are only available for a sparse subset of frames in a video. Our
model is trained end-to-end on color images to jointly reconstruct hands and
objects in 3D by inferring their poses. Given our estimated reconstructions, we
differentiably render the optical flow between pairs of adjacent images and use
it within the network to warp one frame to another. We then apply a
self-supervised photometric loss that relies on the visual consistency between
nearby images. We achieve state-of-the-art results on 3D hand-object
reconstruction benchmarks and demonstrate that our approach allows us to
improve the pose estimation accuracy by leveraging information from neighboring
frames in low-data regimes.

模拟手 - 物互动的姿态估计方法，通过不依赖稠密标注数据的光度一致性方法可同时进行手和物体的三维重建和精确定位，取得了 3D 手 - 物重建基准测试的最优结果，并展示了其在低数据场景下通过相邻帧获取信息以提高姿态估计精度的能力。

利用时间上的光度一致性实现稀疏监督的手 - 物体重建

Leveraging Photometric Consistency over Time for Sparsely Supervised  Hand-Object Reconstruction

Estimating hand-object manipulations is essential for interpreting and
imitating human actions. Previous work has made significant progress towards
reconstruction of hand poses and object shapes in isolation. Yet,
reconstructing hands and objects during manipulation is a more challenging task
due to significant occlusions of both the hand and object. While presenting
challenges, manipulations may also simplify the problem since the physics of
contact restricts the space of valid hand-object configurations. For example,
during manipulation, the hand and object should be in contact but not
interpenetrate. In this work, we regularize the joint reconstruction of hands
and objects with manipulation constraints. We present an end-to-end learnable
model that exploits a novel contact loss that favors physically plausible
hand-object constellations. Our approach improves grasp quality metrics over
baselines, using RGB images as input. To train and evaluate the model, we also
propose a new large-scale synthetic dataset, ObMan, with hand-object
manipulations. We demonstrate the transferability of ObMan-trained models to
real data.

基于物理约束的手 - 物体重构模型，利用合成数据集 ObMan 进行模型训练和验证，达到手 - 物交互场景下的稳定手部姿势和可行物体形状重构。