We present GeoNeRF, a generalizable photorealistic novel view synthesis
method based on neural radiance fields. Our approach consists of two main
stages: a geometry reasoner and a renderer. To render a novel view, the
geometry reasoner first constructs cascaded cost volumes for each nearby source
view. Then, using a Transformer-based attention mechanism and the cascaded cost
volumes, the renderer infers geometry and appearance, and renders detailed
images via classical volume rendering techniques. This architecture, in
particular, allows sophisticated occlusion reasoning, gathering information
from consistent source views. Moreover, our method can easily be fine-tuned on
a single scene, and renders competitive results with per-scene optimized neural
rendering methods with a fraction of computational cost. Experiments show that
GeoNeRF outperforms state-of-the-art generalizable neural rendering models on
various synthetic and real datasets. Lastly, with a slight modification to the
geometry reasoner, we also propose an alternative model that adapts to RGBD
images. This model directly exploits the depth information often available
thanks to depth sensors. The implementation code is available at
this https URL

该论文提出了一种基于神经辐射场的通用逼真新视角合成方法，该方法包括几何推理和渲染两个主要阶段，可用于各种合成和实际数据集。

GeoNeRF：利用几何先验的通用 NeRF

GeoNeRF: Generalizing NeRF with Geometry Priors

In this paper we present, to the best of our knowledge, the first method to
learn a generative model of 3D shapes from natural images in a fully
unsupervised way. For example, we do not use any ground truth 3D or 2D
annotations, stereo video, and ego-motion during the training. Our approach
follows the general strategy of Generative Adversarial Networks, where an image
generator network learns to create image samples that are realistic enough to
fool a discriminator network into believing that they are natural images. In
contrast, in our approach the image generation is split into 2 stages. In the
first stage a generator network outputs 3D objects. In the second, a
differentiable renderer produces an image of the 3D objects from random
viewpoints. The key observation is that a realistic 3D object should yield a
realistic rendering from any plausible viewpoint. Thus, by randomizing the
choice of the viewpoint our proposed training forces the generator network to
learn an interpretable 3D representation disentangled from the viewpoint. In
this work, a 3D representation consists of a triangle mesh and a texture map
that is used to color the triangle surface by using the UV-mapping technique.
We provide analysis of our learning approach, expose its ambiguities and show
how to overcome them. Experimentally, we demonstrate that our method can learn
realistic 3D shapes of faces by using only the natural images of the FFHQ
dataset.

本文介绍了一种从自然图像中学习生成 3D 形状的生成模型的全面无监督方法，其中利用 GAN 等深度学习技术实现，从而生成出真实的三维图像，实现了从二维图像向三维图像的转换。