In this work we introduce Lifting Autoencoders, a generative 3D surface-based
model of object categories. We bring together ideas from non-rigid structure
from motion, image formation, and morphable models to learn a controllable,
geometric model of 3D categories in an entirely unsupervised manner from an
unstructured set of images. We exploit the 3D geometric nature of our model and
use normal information to disentangle appearance into illumination, shading and
albedo. We further use weak supervision to disentangle the non-rigid shape
variability of human faces into identity and expression. We combine the 3D
representation with a differentiable renderer to generate RGB images and append
an adversarially trained refinement network to obtain sharp, photorealistic
image reconstruction results. The learned generative model can be controlled in
terms of interpretable geometry and appearance factors, allowing us to perform
photorealistic image manipulation of identity, expression, 3D pose, and
illumination properties.

介绍了一种新的 3D 表面基础的物体类别生成模型 - Lifting Autoencoders，该模型结合了非刚性结构运动、图像形成和形态模型等的思想，能完全无监督地从非结构化的图像集中学习可控、几何的 3D 类别生成模型，并使用可微渲染器生成 RGB 图像和对抗性训练的细化网络实现清晰逼真的图像重建结果。其生成模型可以通过可识别的几何和外观因素进行控制，从而进行身份、表达、3D 姿势和光照属性的逼真图像操作。