We propose a novel framework for training neural networks which is capable of learning 3D information of non-rigid objects when only 2D annotations are available as ground truths. Recently, there have been some approaches that incorporate the problem setting of non-rigid structure-from-motion (NRSfM) into deep learning to learn 3D structure reconstruction. The most important difficulty of NRSfM is to estimate both the rotation and deformation at the same time, and previous works handle this by regressing both of them. In this paper, we resolve this difficulty by proposing a loss function wherein the suitable rotation is automatically determined. Trained with the cost function consisting of the reprojection error and the low-rank term of aligned shapes, the network learns the 3D structures of such objects as human skeletons and faces during the training, whereas the testing is done in a single-frame basis. The proposed method can handle inputs with missing entries and experimental results validate that the proposed framework shows superior reconstruction performance to the state-of-the-art method on the Human 3.6M, 300-VW, and SURREAL datasets, even though the underlying network structure is very simple.

提出了一种新的神经网络训练框架，可以在只有2D注释的情况下学习非刚性物体的3D信息，该框架通过自动确定适当的旋转解决了非刚性结构运动估计中同时估计旋转和变形的难点，使用损失函数进行训练后，网络可以学习人类骨架和面孔等对象的3D结构，在测试过程中可以处理带有丢失入口的输入，实验证明该框架表现优异，即使底层网络结构非常简单，在Human 3.6M、300-VW和SURREAL数据集上均优于现有最先进的方法。

Procrustes 回归网络: 从二维标注中学习非刚性物体的三维结构