We cast multiview reconstruction from unknown pose as a generative modeling
problem. From a collection of unannotated 2D images of a scene, our approach
simultaneously learns both a network to predict camera pose from 2D image
input, as well as the parameters of a Neural Radiance Field (NeRF) for the 3D
scene. To drive learning, we wrap both the pose prediction network and NeRF
inside a Denoising Diffusion Probabilistic Model (DDPM) and train the system
via the standard denoising objective. Our framework requires the system
accomplish the task of denoising an input 2D image by predicting its pose and
rendering the NeRF from that pose. Learning to denoise thus forces the system
to concurrently learn the underlying 3D NeRF representation and a mapping from
images to camera extrinsic parameters. To facilitate the latter, we design a
custom network architecture to represent pose as a distribution, granting
implicit capacity for discovering view correspondences when trained end-to-end
for denoising alone. This technique allows our system to successfully build
NeRFs, without pose knowledge, for challenging scenes where competing methods
fail. At the conclusion of training, our learned NeRF can be extracted and used
as a 3D scene model; our full system can be used to sample novel camera poses
and generate novel-view images.

我们将来自未知姿态的多视图重建问题作为一个生成建模问题。通过一组未标注的场景的 2D 图像，我们的方法同时学习了一个网络来预测从 2D 图像输入中的相机姿态，并学习了用于 3D 场景的神经辐射场（NeRF）的参数。通过在标准去噪目标下使用去噪扩散概率模型（DDPM）将姿态预测网络和 NeRF 结合在一起并训练系统来推动学习进展。我们的框架要求系统通过预测图像的姿态并渲染 NeRF 来完成去噪输入 2D 图像的任务。学习去噪因此迫使系统同时学习底层的 3D NeRF 表示和从图像到摄像机外参数的映射。为了促进后者，我们设计了一个自定义的网络架构来将姿态表示为一个分布，从而在仅进行去噪训练时具备发现视图对应关系的隐式能力。这种技术允许我们的系统成功构建 NeRFs，并在竞争方法失败的具有挑战性的场景中使用。在训练结束时，我们学习到的 NeRF 可以被提取并用作 3D 场景模型；我们的完整系统可以用于采样新的相机姿态并生成新的视角图像。