Real-time tracking of 3D hand pose in world space is a challenging problem
and plays an important role in VR interaction. Existing work in this space are
limited to either producing root-relative (versus world space) 3D pose or rely
on multiple stages such as generating heatmaps and kinematic optimization to
obtain 3D pose. Moreover, the typical VR scenario, which involves multi-view
tracking from wide \ac{fov} cameras is seldom addressed by these methods. In
this paper, we present a unified end-to-end differentiable framework for
multi-view, multi-frame hand tracking that directly predicts 3D hand pose in
world space. We demonstrate the benefits of end-to-end differentiabilty by
extending our framework with downstream tasks such as jitter reduction and
pinch prediction. To demonstrate the efficacy of our model, we further present
a new large-scale egocentric hand pose dataset that consists of both real and
synthetic data. Experiments show that our system trained on this dataset
handles various challenging interactive motions, and has been successfully
applied to real-time VR applications.

本文提出了一种多视角、多帧手部跟踪的统一端到端可微框架，直接预测世界空间中的三维手部姿态，并演示了将框架扩展到降噪和捏预测等下游任务的好处。实验证明，该系统在处理各种具有挑战性的交互动作方面表现出色，并已成功应用于实时虚拟现实应用。