We present a method for recovering the dense 3D surface of the hand by
regressing the vertex coordinates of a mesh model from a single depth map. To
this end, we use a two-stage 2D fully convolutional network architecture. In
the first stage, the network estimates a dense correspondence field for every
pixel on the depth map or image grid to the mesh grid. In the second stage, we
design a differentiable operator to map features learned from the previous
stage and regress a 3D coordinate map on the mesh grid. Finally, we sample from
the mesh grid to recover the mesh vertices, and fit it an articulated template
mesh in closed form. During inference, the network can predict all the mesh
vertices, transformation matrices for every joint and the joint coordinates in
a single forward pass. When given supervision on the sparse key-point
coordinates, our method achieves state-of-the-art accuracy on NYU dataset for
key point localization while recovering mesh vertices and a dense
correspondence map. Our framework can also be learned through self-supervision
by minimizing a set of data fitting and kinematic prior terms. With
multi-camera rig during training to resolve self-occlusion, it can perform
competitively with strongly supervised methods Without any human annotation.

本文提出了一种通过从单个深度图中回归网格模型顶点坐标来恢复手部密集 3D 表面的方法，使用两阶段 2D 完全卷积网络架构和可微分算子来实现，能够通过自我监督实现更好的效果，表现优异。