3D hand tracking from a monocular video is a very challenging problem due to
hand interactions, occlusions, left-right hand ambiguity, and fast motion. Most
existing methods rely on RGB inputs, which have severe limitations under
low-light conditions and suffer from motion blur. In contrast, event cameras
capture local brightness changes instead of full image frames and do not suffer
from the described effects. Unfortunately, existing image-based techniques
cannot be directly applied to events due to significant differences in the data
modalities. In response to these challenges, this paper introduces the first
framework for 3D tracking of two fast-moving and interacting hands from a
single monocular event camera. Our approach tackles the left-right hand
ambiguity with a novel semi-supervised feature-wise attention mechanism and
integrates an intersection loss to fix hand collisions. To facilitate advances
in this research domain, we release a new synthetic large-scale dataset of two
interacting hands, Ev2Hands-S, and a new real benchmark with real event streams
and ground-truth 3D annotations, Ev2Hands-R. Our approach outperforms existing
methods in terms of the 3D reconstruction accuracy and generalises to real data
under severe light conditions.

提出了首个从单目事件摄像机中跟踪两只快速移动和互动的手的三维跟踪框架，通过新颖的半监督特征注意机制解决左右手歧义并整合相交损失来修复手的碰撞，推出了新的大规模数据集 Ev2Hands-S 和真实事件流与真实三维标注的基准数据集 Ev2Hands-R，并在强光条件下对真实数据具有更高的三维重建精度。

单目事件相机下两只交互手的 3D 姿态估计

3D Pose Estimation of Two Interacting Hands from a Monocular Event  Camera

We address the highly challenging problem of real-time 3D hand tracking based
on a monocular RGB-only sequence. Our tracking method combines a convolutional
neural network with a kinematic 3D hand model, such that it generalizes well to
unseen data, is robust to occlusions and varying camera viewpoints, and leads
to anatomically plausible as well as temporally smooth hand motions. For
training our CNN we propose a novel approach for the synthetic generation of
training data that is based on a geometrically consistent image-to-image
translation network. To be more specific, we use a neural network that
translates synthetic images to "real" images, such that the so-generated images
follow the same statistical distribution as real-world hand images. For
training this translation network we combine an adversarial loss and a
cycle-consistency loss with a geometric consistency loss in order to preserve
geometric properties (such as hand pose) during translation. We demonstrate
that our hand tracking system outperforms the current state-of-the-art on
challenging RGB-only footage.

本研究介绍了一种基于单目 RGB 彩色序列的实时 3D 手部追踪的方法，其结合了卷积神经网络和运动学 3D 手模型，提出了一种基于几何一致的图像到图像翻译网络的新方法来合成训练数据，并在拥有遮挡和不同相机视角的情况下进行了实验，该方法在 RGB 序列跟踪中优于当前最先进的技术。