We propose the NeRF-LEBM, a likelihood-based top-down 3D-aware 2D image
generative model that incorporates 3D representation via Neural Radiance Fields
(NeRF) and 2D imaging process via differentiable volume rendering. The model
represents an image as a rendering process from 3D object to 2D image and is
conditioned on some latent variables that account for object characteristics
and are assumed to follow informative trainable energy-based prior models. We
propose two likelihood-based learning frameworks to train the NeRF-LEBM: (i)
maximum likelihood estimation with Markov chain Monte Carlo-based inference and
(ii) variational inference with the reparameterization trick. We study our
models in the scenarios with both known and unknown camera poses. Experiments
on several benchmark datasets demonstrate that the NeRF-LEBM can infer 3D
object structures from 2D images, generate 2D images with novel views and
objects, learn from incomplete 2D images, and learn from 2D images with known
or unknown camera poses.

该论文提出了一种基于可能性的自顶向下三维感知二维图像生成模型 NeRF-LEBM，它通过神经辐射场 (NeRF) 同时考虑了三维表示和二维成像过程，并通过可微分体积渲染来实现。作者提出了两种基于可能性的学习框架用于训练 NeRF-LEBM，并对已知和未知相机位姿的情况下的模型进行了研究。实验表明，NeRF-LEBM 能够从二维图像推断出三维物体结构，并生成具有新视角和新物体的二维图像。