We propose NViST, a transformer-based model for novel-view synthesis from a
single image, trained on a large-scale dataset of in-the-wild images with
complex backgrounds. NViST transforms image inputs directly into a radiance
field, adopting a scalable transformer-based architecture. In practice, NViST
exploits the self-supervised features learnt by a masked autoencoder (MAE), and
learns a novel decoder that translates features to 3D tokens via
cross-attention and adaptive layer normalization. Our model is efficient at
inference since only a single forward-pass is needed to predict a 3D
representation, unlike methods that require test-time optimization or sampling
such as 3D-aware diffusion models. We tackle further limitations of current
new-view synthesis models. First, unlike most generative models that are
trained in a category-specific manner, often on synthetic datasets or on masked
inputs, our model is trained on MVImgNet, a large-scale dataset of real-world,
casually-captured videos containing hundreds of object categories with diverse
backgrounds. Secondly, our model does not require canonicalization of the
training data - i.e. aligning all objects with a frontal view - only needing
relative pose at training time which removes a substantial barrier to it being
used on casually captured datasets. We show results on unseen objects and
categories on MVImgNet and even casual phone captures. We conduct qualitative
and quantitative evaluations on MVImgNet and ShapeNet to show that our model
represents a step forward towards enabling true in-the-wild novel-view
synthesis from a single image.

我们提出了一种基于 transformer 的模型 NViST，用于从单张图像中合成新视角，该模型在具有复杂背景的大规模野外图像数据集上进行训练。