In this paper, we introduce 4DHands, a robust approach to recovering
interactive hand meshes and their relative movement from monocular inputs. Our
approach addresses two major limitations of previous methods: lacking a unified
solution for handling various hand image inputs and neglecting the positional
relationship of two hands within images. To overcome these challenges, we
develop a transformer-based architecture with novel tokenization and feature
fusion strategies. Specifically, we propose a Relation-aware Two-Hand
Tokenization (RAT) method to embed positional relation information into the
hand tokens. In this way, our network can handle both single-hand and two-hand
inputs and explicitly leverage relative hand positions, facilitating the
reconstruction of intricate hand interactions in real-world scenarios. As such
tokenization indicates the relative relationship of two hands, it also supports
more effective feature fusion. To this end, we further develop a
Spatio-temporal Interaction Reasoning (SIR) module to fuse hand tokens in 4D
with attention and decode them into 3D hand meshes and relative temporal
movements. The efficacy of our approach is validated on several benchmark
datasets. The results on in-the-wild videos and real-world scenarios
demonstrate the superior performances of our approach for interactive hand
reconstruction. More video results can be found on the project page:
this https URL

通过 4DHands 方法，从单目输入中恢复互动手部网格及其相对运动，处理了自由手图像输入和两只手的位置关系的限制，并通过新颖的分词和特征融合策略提出了一种基于 Transformer 的架构。

4DHands: 使用 Transformer 重建 4D 交互手势

4DHands: Reconstructing Interactive Hands in 4D with Transformers

Perceiving 3D objects from monocular inputs is crucial for robotic systems,
given its economy compared to multi-sensor settings. It is notably difficult as
a single image can not provide any clues for predicting absolute depth values.
Motivated by binocular methods for 3D object detection, we take advantage of
the strong geometry structure provided by camera ego-motion for accurate object
depth estimation and detection. We first make a theoretical analysis on this
general two-view case and notice two challenges: 1) Cumulative errors from
multiple estimations that make the direct prediction intractable; 2) Inherent
dilemmas caused by static cameras and matching ambiguity. Accordingly, we
establish the stereo correspondence with a geometry-aware cost volume as the
alternative for depth estimation and further compensate it with monocular
understanding to address the second problem. Our framework, named Depth from
Motion (DfM), then uses the established geometry to lift 2D image features to
the 3D space and detects 3D objects thereon. We also present a pose-free DfM to
make it usable when the camera pose is unavailable. Our framework outperforms
state-of-the-art methods by a large margin on the KITTI benchmark. Detailed
quantitative and qualitative analyses also validate our theoretical
conclusions. The code will be released at
this https URL

通过研究基于单目输入的三维物体感知，提出了一种利用相机自运动提供的强几何结构进行精确物体深度估计和检测的方法，并建立了一个名为 “DfM” 的框架，成功地提高了 KITTI 基准数据集上的检测性能。