3D hand shape and pose estimation from a single depth map is a new and
challenging computer vision problem with many applications. The
state-of-the-art methods directly regress 3D hand meshes from 2D depth images
via 2D convolutional neural networks, which leads to artefacts in the
estimations due to perspective distortions in the images. In contrast, we
propose a novel architecture with 3D convolutions trained in a
weakly-supervised manner. The input to our method is a 3D voxelized depth map,
and we rely on two hand shape representations. The first one is the 3D
voxelized grid of the shape which is accurate but does not preserve the mesh
topology and the number of mesh vertices. The second representation is the 3D
hand surface which is less accurate but does not suffer from the limitations of
the first representation. We combine the advantages of these two
representations by registering the hand surface to the voxelized hand shape. In
the extensive experiments, the proposed approach improves over the state of the
art by 47.8% on the SynHand5M dataset. Moreover, our augmentation policy for
voxelized depth maps further enhances the accuracy of 3D hand pose estimation
on real data. Our method produces visually more reasonable and realistic hand
shapes on NYU and BigHand2.2M datasets compared to the existing approaches.

通过弱监督学习的 3D 卷积网络结构，结合两种不同的手部形状表示，将手部表面映射到体素化手型图形之中，得到比现有手部姿态估计方法更加准确的结果，并在真实数据上通过体素化深度图扩充策略使得模型更具泛化性能。

HandVoxNet：一种基于深度体素网络的单深度图像三维手部形态与姿态估计方法

HandVoxNet: Deep Voxel-Based Network for 3D Hand Shape and Pose  Estimation from a Single Depth Map

This work addresses a novel and challenging problem of estimating the full 3D
hand shape and pose from a single RGB image. Most current methods in 3D hand
analysis from monocular RGB images only focus on estimating the 3D locations of
hand keypoints, which cannot fully express the 3D shape of hand. In contrast,
we propose a Graph Convolutional Neural Network (Graph CNN) based method to
reconstruct a full 3D mesh of hand surface that contains richer information of
both 3D hand shape and pose. To train networks with full supervision, we create
a large-scale synthetic dataset containing both ground truth 3D meshes and 3D
poses. When fine-tuning the networks on real-world datasets without 3D ground
truth, we propose a weakly-supervised approach by leveraging the depth map as a
weak supervision in training. Through extensive evaluations on our proposed new
datasets and two public datasets, we show that our proposed method can produce
accurate and reasonable 3D hand mesh, and can achieve superior 3D hand pose
estimation accuracy when compared with state-of-the-art methods.

本文提出了一种基于图形卷积神经网络的方法，从单个 RGB 图像中估算手的 3D 形状和姿态，并通过含 3D 地面真值的大规模合成数据集和深度图弱监督学习进行训练，结果表明可以对手的 3D 网格进行准确和合理的重建，并在与现有最先进方法的比较中达到了更高的 3D 姿态估计准确性。

从单一 RGB 图像中估计 3D 手型和姿势

3D Hand Shape and Pose Estimation from a Single RGB Image

We present in this work the first end-to-end deep learning based method that
predicts both 3D hand shape and pose from RGB images in the wild. Our network
consists of the concatenation of a deep convolutional encoder, and a fixed
model-based decoder. Given an input image, and optionally 2D joint detections
obtained from an independent CNN, the encoder predicts a set of hand and view
parameters. The decoder has two components: A pre-computed articulated mesh
deformation hand model that generates a 3D mesh from the hand parameters, and a
re-projection module controlled by the view parameters that projects the
generated hand into the image domain. We show that using the shape and pose
prior knowledge encoded in the hand model within a deep learning framework
yields state-of-the-art performance in 3D pose prediction from images on
standard benchmarks, and produces geometrically valid and plausible 3D
reconstructions. Additionally, we show that training with weak supervision in
the form of 2D joint annotations on datasets of images in the wild, in
conjunction with full supervision in the form of 3D joint annotations on
limited available datasets allows for good generalization to 3D shape and pose
predictions on images in the wild.

利用先前计算好的人工建模和先验知识，结合深度学习的方法实现了从 RGB 图像中预测手部形状和姿态，结果显示出在标准基准测试中表现出最新颖的 3D 姿态预测效果，同时在弱监督下训练的模型也能够很好的应用于在实际环境中的 3D 形状和姿态预测。