Applying data-driven approaches to non-rigid 3D reconstruction has been difficult, which we believe can be attributed to the lack of a large-scale training corpus. One recent approach proposes self-supervision based on non-rigid reconstruction. Unfortunately, this method fails for important cases such as highly non-rigid deformations. We first address this problem of lack of data by introducing a novel semi-supervised strategy to obtain dense inter-frame correspondences from a sparse set of annotations. This way, we obtain a large dataset of 400 scenes, over 390,000 RGB-D frames, and 2,537 densely aligned frame pairs; in addition, we provide a test set along with several metrics for evaluation. Based on this corpus, we introduce a data-driven non-rigid feature matching approach, which we integrate into an optimization-based reconstruction pipeline. Here, we propose a new neural network that operates on RGB-D frames, while maintaining robustness under large non-rigid deformations and producing accurate predictions. Our approach significantly outperforms both existing non-rigid reconstruction methods that do not use learned data terms, as well as learning-based approaches that only use self-supervision.

利用数据驱动的方法进行三维重建非常困难，本文通过引入新的半监督策略，成功建立了一个拥有 400 个场景、超过 390,000 个 RGB-D 帧和 5,533 个面对面对齐帧对的数据集，然后利用这个数据集，提出了一个能够应对非常大的非刚性形变的深度学习方法，对比于其他方法，我们的结果优于现有的方法。

DeepDeform: 半监督学习非刚性RGB-D重建