Recent work has shown the ability to learn generative models for 3D shapes
from only unstructured 2D images. However, training such models requires
differentiating through the rasterization step of the rendering process,
therefore past work has focused on developing bespoke rendering models which
smooth over this non-differentiable process in various ways. Such models are
thus unable to take advantage of the photo-realistic, fully featured,
industrial renderers built by the gaming and graphics industry. In this paper
we introduce the first scalable training technique for 3D generative models
from 2D data which utilizes an off-the-shelf non-differentiable renderer. To
account for the non-differentiability, we introduce a proxy neural renderer to
match the output of the non-differentiable renderer. We further propose
discriminator output matching to ensure that the neural renderer learns to
smooth over the rasterization appropriately. We evaluate our model on images
rendered from our generated 3D shapes, and show that our model can consistently
learn to generate better shapes than existing models when trained with
exclusively unstructured 2D images.

本文提出了一种使用通用的不可微分渲染器训练来自 2D 数据的 3D 生成模型的可扩展方法，该方法引入了一个代理神经渲染器来匹配不可微分渲染器的输出。我们进一步提出鉴别器输出匹配以确保神经渲染器学习适当地平滑栅格化。当用纯 2D 图像进行训练时，我们的模型可以比现有模型更好地学习生成更好的 3D 形状。