We study the problem of learning to estimate the 3D object pose from a few
labelled examples and a collection of unlabelled data. Our main contribution is
a learning framework, neural view synthesis and matching, that can transfer the
3D pose annotation from the labelled to unlabelled images reliably, despite
unseen 3D views and nuisance variations such as the object shape, texture,
illumination or scene context. In our approach, objects are represented as 3D
cuboid meshes composed of feature vectors at each mesh vertex. The model is
initialized from a few labelled images and is subsequently used to synthesize
feature representations of unseen 3D views. The synthesized views are matched
with the feature representations of unlabelled images to generate pseudo-labels
of the 3D pose. The pseudo-labelled data is, in turn, used to train the feature
extractor such that the features at each mesh vertex are more invariant across
varying 3D views of the object. Our model is trained in an EM-type manner
alternating between increasing the 3D pose invariance of the feature extractor
and annotating unlabelled data through neural view synthesis and matching. We
demonstrate the effectiveness of the proposed semi-supervised learning
framework for 3D pose estimation on the PASCAL3D+ and KITTI datasets. We find
that our approach outperforms all baselines by a wide margin, particularly in
an extreme few-shot setting where only 7 annotated images are given.
Remarkably, we observe that our model also achieves an exceptional robustness
in out-of-distribution scenarios that involve partial occlusion.

本文提出了一个基于神经网络视图合成和匹配的学习框架，通过生成的伪标签来学习众多未标注图像中的物体 3D 姿态估计，同时结合 EM 算法的方式，逐步提高特征提取器在不同 3D 视角下的不变性，并通过在 PASCAL3D+ 和 KITTI 数据集上进行的实验证明，该方法在少样本学习下，尤其是掩盖度极高的情况下，比其他基线方法表现更好，且具有出色的鲁棒性。