Recent works in hand-object reconstruction mainly focus on the single-view
and dense multi-view settings. On the one hand, single-view methods can
leverage learned shape priors to generalise to unseen objects but are prone to
inaccuracies due to occlusions. On the other hand, dense multi-view methods are
very accurate but cannot easily adapt to unseen objects without further data
collection. In contrast, sparse multi-view methods can take advantage of the
additional views to tackle occlusion, while keeping the computational cost low
compared to dense multi-view methods. In this paper, we consider the problem of
hand-object reconstruction with unseen objects in the sparse multi-view
setting. Given multiple RGB images of the hand and object captured at the same
time, our model SVHO combines the predictions from each view into a unified
reconstruction without optimisation across views. We train our model on a
synthetic hand-object dataset and evaluate directly on a real world recorded
hand-object dataset with unseen objects. We show that while reconstruction of
unseen hands and objects from RGB is challenging, additional views can help
improve the reconstruction quality.

在稀疏多视图设置中，我们考虑了手 - 物体重建问题，给定同时捕获的手和物体的多个 RGB 图像，我们的模型 SVHO 将每个视图的预测融合成统一的重建，通过在合成手 - 物体数据集上训练，并在真实世界记录的未见过的物体的手 - 物体数据集上进行直接评估，我们显示出附加视图可以提高重建质量。

用于未知环境的稀疏多视图手 - 物体重建

Sparse multi-view hand-object reconstruction for unseen environments

We address in-the-wild hand-object reconstruction for a known object category
in egocentric videos, focusing on temporal periods of stable grasps. We propose
the task of Hand-Object Stable Grasp Reconstruction (HO-SGR), the joint
reconstruction of frames during which the hand is stably holding the object. We
thus can constrain the object motion relative to the hand, effectively
regularising the reconstruction and improving performance. By analysing the 3D
ARCTIC dataset, we identify temporal periods where the contact area between the
hand and object vertices remain stable. We showcase that objects within stable
grasps move within a single degree of freedom (1~DoF). We thus propose a method
for jointly optimising all frames within a stable grasp by minimising the
object's rotation to that within a latent 1 DoF. We then extend this knowledge
to in-the-wild egocentric videos by labelling 2.4K clips of stable grasps from
the EPIC-KITCHENS dataset. Our proposed EPIC-Grasps dataset includes 390 object
instances of 9 categories, featuring stable grasps from videos of daily
interactions in 141 environments. Our method achieves significantly better
HO-SGR, both qualitatively and by computing the stable grasp area and 2D
projection labels of mask overlaps.

我们提出了一种在日常环境下进行手 - 物体重建的方法，通过对稳定握持期间的帧进行联合重建，约束物体运动并提高重建性能。我们通过分析 3D ARCTIC 数据集，确定了手和物体之间接触区域保持稳定的时间段，并用实验展示了稳定握持期间物体仅在一个自由度内运动。然后，我们将这一方法扩展到真实环境中的 egocentric 视频，通过标记 EPIC-KITCHENS 数据集中稳定握持的剪辑，建立了 EPIC-Grasps 数据集，其中包括了 141 个环境中的 390 个物体实例，分为 9 个类别，并通过计算稳定握持区域和 2D 投影标签的掩码重叠，定量地展示了我们方法在稳定握持重建方面的显著改进。

握紧：在自视角视频中重建手 - 物体稳定握持

Get a Grip: Reconstructing Hand-Object Stable Grasps in Egocentric  Videos

Reconstructing both objects and hands in 3D from a single RGB image is
complex. Existing methods rely on manually defined hand-object constraints in
Euclidean space, leading to suboptimal feature learning. Compared with
Euclidean space, hyperbolic space better preserves the geometric properties of
meshes thanks to its exponentially-growing space distance, which amplifies the
differences between the features based on similarity. In this work, we propose
the first precise hand-object reconstruction method in hyperbolic space, namely
Dynamic Hyperbolic Attention Network (DHANet), which leverages intrinsic
properties of hyperbolic space to learn representative features. Our method
that projects mesh and image features into a unified hyperbolic space includes
two modules, ie. dynamic hyperbolic graph convolution and image-attention
hyperbolic graph convolution. With these two modules, our method learns mesh
features with rich geometry-image multi-modal information and models better
hand-object interaction. Our method provides a promising alternative for fine
hand-object reconstruction in hyperbolic space. Extensive experiments on three
public datasets demonstrate that our method outperforms most state-of-the-art
methods.

本文提出了首个在双曲空间中精确手物体重构的方法 Dynamic Hyperbolic Attention Network (DHANet)，该方法借助双曲空间的内在特性学习代表性特征，并通过动态双曲图卷积和基于图注意力的双曲图卷积两个模块来学习具有丰富几何 - 图像多模态信息的网格特征和更好地建模手物体交互作用，实验证明该方法在三个公共数据集上优于大多数最先进的方法。