We cast multiview reconstruction from unknown pose as a generative modeling
problem. From a collection of unannotated 2D images of a scene, our approach
simultaneously learns both a network to predict camera pose from 2D image
input, as well as the parameters of a Neural Radiance Field (NeRF) for the 3D
scene. To drive learning, we wrap both the pose prediction network and NeRF
inside a Denoising Diffusion Probabilistic Model (DDPM) and train the system
via the standard denoising objective. Our framework requires the system
accomplish the task of denoising an input 2D image by predicting its pose and
rendering the NeRF from that pose. Learning to denoise thus forces the system
to concurrently learn the underlying 3D NeRF representation and a mapping from
images to camera extrinsic parameters. To facilitate the latter, we design a
custom network architecture to represent pose as a distribution, granting
implicit capacity for discovering view correspondences when trained end-to-end
for denoising alone. This technique allows our system to successfully build
NeRFs, without pose knowledge, for challenging scenes where competing methods
fail. At the conclusion of training, our learned NeRF can be extracted and used
as a 3D scene model; our full system can be used to sample novel camera poses
and generate novel-view images.

我们将来自未知姿态的多视图重建问题作为一个生成建模问题。通过一组未标注的场景的 2D 图像，我们的方法同时学习了一个网络来预测从 2D 图像输入中的相机姿态，并学习了用于 3D 场景的神经辐射场（NeRF）的参数。通过在标准去噪目标下使用去噪扩散概率模型（DDPM）将姿态预测网络和 NeRF 结合在一起并训练系统来推动学习进展。我们的框架要求系统通过预测图像的姿态并渲染 NeRF 来完成去噪输入 2D 图像的任务。学习去噪因此迫使系统同时学习底层的 3D NeRF 表示和从图像到摄像机外参数的映射。为了促进后者，我们设计了一个自定义的网络架构来将姿态表示为一个分布，从而在仅进行去噪训练时具备发现视图对应关系的隐式能力。这种技术允许我们的系统成功构建 NeRFs，并在竞争方法失败的具有挑战性的场景中使用。在训练结束时，我们学习到的 NeRF 可以被提取并用作 3D 场景模型；我们的完整系统可以用于采样新的相机姿态并生成新的视角图像。

未知姿态下的多视角到 3D 的生成性提升：将 NeRF 包装在扩散中

Generative Lifting of Multiview to 3D from Unknown Pose: Wrapping NeRF  inside Diffusion

We present a neural rendering-based method called NeRO for reconstructing the
geometry and the BRDF of reflective objects from multiview images captured in
an unknown environment. Multiview reconstruction of reflective objects is
extremely challenging because specular reflections are view-dependent and thus
violate the multiview consistency, which is the cornerstone for most multiview
reconstruction methods. Recent neural rendering techniques can model the
interaction between environment lights and the object surfaces to fit the
view-dependent reflections, thus making it possible to reconstruct reflective
objects from multiview images. However, accurately modeling environment lights
in the neural rendering is intractable, especially when the geometry is
unknown. Most existing neural rendering methods, which can model environment
lights, only consider direct lights and rely on object masks to reconstruct
objects with weak specular reflections. Therefore, these methods fail to
reconstruct reflective objects, especially when the object mask is not
available and the object is illuminated by indirect lights. We propose a
two-step approach to tackle this problem. First, by applying the split-sum
approximation and the integrated directional encoding to approximate the
shading effects of both direct and indirect lights, we are able to accurately
reconstruct the geometry of reflective objects without any object masks. Then,
with the object geometry fixed, we use more accurate sampling to recover the
environment lights and the BRDF of the object. Extensive experiments
demonstrate that our method is capable of accurately reconstructing the
geometry and the BRDF of reflective objects from only posed RGB images without
knowing the environment lights and the object masks. Codes and datasets are
available at this https URL

该研究提出了一种称为 NeRO 的基于神经渲染的方法，用于从采集于未知环境中的多视角图像中，重建反射物体的几何和 BRDF。该方法通过两个步骤解决了在多视角情况下反射物体重建的困难，并成功地重建出了几何和 BRDF。