We present a convolutional network capable of generating images of a previously unseen object from arbitrary viewpoints given a single image of this object. The input to the network is a single image and the desired new viewpoint; the output is a view of the object from this desired viewpoint. The network is trained on renderings of synthetic 3D models. It learns an implicit 3D representation of the object class, which allows it to transfer shape knowledge from training instances to a new object instance. Beside the color image, the network can also generate the depth map of an object from arbitrary viewpoints. This allows us to predict 3D point clouds from a single image, which can be fused into a surface mesh. We experimented with cars and chairs. Even though the network is trained on artificial data, it generalizes well to objects in natural images without any modifications.

本文提出了一种卷积网络，可以仅仅基于单张图像预测出物体的RGB图和深度图，并将多幅深度图融合成完整点云后可转换成表面网格来对任意视野下的三维物体进行识别。网络通过合成的三维车和椅子模型进行训练，在有杂乱背景并搭配真实车辆图像的情况下，仍可生成合理的推测。

卷积网络的单张图像多视角3D模型