We propose pixelNeRF, a learning framework that predicts a continuous neural
scene representation conditioned on one or few input images. The existing
approach for constructing neural radiance fields involves optimizing the
representation to every scene independently, requiring many calibrated views
and significant compute time. We take a step towards resolving these
shortcomings by introducing an architecture that conditions a NeRF on image
inputs in a fully convolutional manner. This allows the network to be trained
across multiple scenes to learn a scene prior, enabling it to perform novel
view synthesis in a feed-forward manner from a sparse set of views (as few as
one). Leveraging the volume rendering approach of NeRF, our model can be
trained directly from images with no explicit 3D supervision. We conduct
extensive experiments on ShapeNet benchmarks for single image novel view
synthesis tasks with held-out objects as well as entire unseen categories. We
further demonstrate the flexibility of pixelNeRF by demonstrating it on
multi-object ShapeNet scenes and real scenes from the DTU dataset. In all
cases, pixelNeRF outperforms current state-of-the-art baselines for novel view
synthesis and single image 3D reconstruction. For the video and code, please
visit the project website: this https URL

我们提出了一种名为 pixelNeRF 的学习框架，可以在仅有一张或几张输入图像的条件下预测连续的神经场景表示。与现有方法相比，我们的体系结构不需要每个场景都进行优化，从而使得在仅有一张图像的情况下就可以执行视角的合成和单幅图像的三维重建。经过大量的实验证明，我们的模型在形状、类别和实景等方面都优于现有的最先进技术。