Novel view synthesis is a long-standing problem. In this work, we consider a
variant of the problem where we are given only a few context views sparsely
covering a scene or an object. The goal is to predict novel viewpoints in the
scene, which requires learning priors. The current state of the art is based on
Neural Radiance Field (NeRF), and while achieving impressive results, the
methods suffer from long training times as they require evaluating millions of
3D point samples via a neural network for each image. We propose a 2D-only
method that maps multiple context views and a query pose to a new image in a
single pass of a neural network. Our model uses a two-stage architecture
consisting of a codebook and a transformer model. The codebook is used to embed
individual images into a smaller latent space, and the transformer solves the
view synthesis task in this more compact space. To train our model efficiently,
we introduce a novel branching attention mechanism that allows us to use the
same model not only for neural rendering but also for camera pose estimation.
Experimental results on real-world scenes show that our approach is competitive
compared to NeRF-based methods while not reasoning explicitly in 3D, and it is
faster to train.

本研究提出了一种基于 2D 的神经网络方法，结合编码键和转换模型，用于实现新颖视角合成，在不需要显式 3D 理解的情况下，相比于 NeRF 方法更高效。