Novel view synthesis aims to generate new view images of a given view image
collection. Recent attempts address this problem relying on 3D geometry priors
(e.g., shapes, sizes, and positions) learned from multi-view images. However,
such methods encounter the following limitations: 1) they require a set of
multi-view images as training data for a specific scene (e.g., face, car or
chair), which is often unavailable in many real-world scenarios; 2) they fail
to extract the geometry priors from single-view images due to the lack of
multi-view supervision. In this paper, we propose a Geometry-enhanced NeRF
(G-NeRF), which seeks to enhance the geometry priors by a geometry-guided
multi-view synthesis approach, followed by a depth-aware training. In the
synthesis process, inspired that existing 3D GAN models can unconditionally
synthesize high-fidelity multi-view images, we seek to adopt off-the-shelf 3D
GAN models, such as EG3D, as a free source to provide geometry priors through
synthesizing multi-view data. Simultaneously, to further improve the geometry
quality of the synthetic data, we introduce a truncation method to effectively
sample latent codes within 3D GAN models. To tackle the absence of multi-view
supervision for single-view images, we design the depth-aware training
approach, incorporating a depth-aware discriminator to guide geometry priors
through depth maps. Experiments demonstrate the effectiveness of our method in
terms of both qualitative and quantitative results.

通过使用 Geometry-enhanced NeRF (G-NeRF) 方法，在 novel view synthesis 中增强几何先验，通过几何引导的多视角合成方法和深度感知型训练，解决了多视角数据不足和单视角图像几何先验提取问题，并通过定剪方法和深度感知鉴别器进一步提高了几何品质。实验证明了我们方法在定量和定性结果上的有效性。

G-NeRF: 单视图图像的几何增强新视角合成

G-NeRF: Geometry-enhanced Novel View Synthesis from Single-View Images

The Abstraction and Reasoning Corpus (ARC) (Chollet, 2019) and its most
recent language-complete instantiation (LARC) has been postulated as an
important step towards general AI. Yet, even state-of-the-art machine learning
models struggle to achieve meaningful performance on these problems, falling
behind non-learning based approaches. We argue that solving these tasks
requires extreme generalization that can only be achieved by proper accounting
for core knowledge priors. As a step towards this goal, we focus on geometry
priors and introduce LatFormer, a model that incorporates lattice symmetry
priors in attention masks. We show that, for any transformation of the
hypercubic lattice, there exists a binary attention mask that implements that
group action. Hence, our study motivates a modification to the standard
attention mechanism, where attention weights are scaled using soft masks
generated by a convolutional network. Experiments on synthetic geometric
reasoning show that LatFormer requires 2 orders of magnitude fewer data than
standard attention and transformers. Moreover, our results on ARC and LARC
tasks that incorporate geometric priors provide preliminary evidence that these
complex datasets do not lie out of the reach of deep learning models.

本文提出了 LatFormer 模型，将晶格对称乘法先验纳入注意力掩码中，以实现极端泛化，并在合成几何推理方面进行了实验证明，表明 LatFormer 需要比标准注意力和转换器少 2 个数量级的数据，并且对于包括几何先验的 ARC 和 LARC 任务的结果提供了初步证据，表明这些复杂的数据集不在深度学习模型的能力之外。

将晶格对称性先验注入注意力机制中，用于提高抽象几何推理的样本效率

Infusing Lattice Symmetry Priors in Attention Mechanisms for  Sample-Efficient Abstract Geometric Reasoning

We present a diffusion-based model for 3D-aware generative novel view
synthesis from as few as a single input image. Our model samples from the
distribution of possible renderings consistent with the input and, even in the
presence of ambiguity, is capable of rendering diverse and plausible novel
views. To achieve this, our method makes use of existing 2D diffusion backbones
but, crucially, incorporates geometry priors in the form of a 3D feature
volume. This latent feature field captures the distribution over possible scene
representations and improves our method's ability to generate view-consistent
novel renderings. In addition to generating novel views, our method has the
ability to autoregressively synthesize 3D-consistent sequences. We demonstrate
state-of-the-art results on synthetic renderings and room-scale scenes; we also
show compelling results for challenging, real-world objects.

使用基于扩散的模型，结合现有的二维扩散骨架和三维特征体，进行三维感知的图像生成，同时具备自回归生成 3D 一致的序列能力。在合成渲染图像和实际物体上展示了最先进的效果。

使用 3D 感知扩散模型实现生成新视图综合

Generative Novel View Synthesis with 3D-Aware Diffusion Models

We present a method that accelerates reconstruction of 3D scenes and objects,
aiming to enable instant reconstruction on edge devices such as mobile phones
and AR/VR headsets. While recent works have accelerated scene reconstruction
training to minute/second-level on high-end GPUs, there is still a large gap to
the goal of instant training on edge devices which is yet highly desired in
many emerging applications such as immersive AR/VR. To this end, this work aims
to further accelerate training by leveraging geometry priors of the target
scene. Our method proposes strategies to alleviate the noise of the imperfect
geometry priors to accelerate the training speed on top of the highly optimized
Instant-NGP. On the NeRF Synthetic dataset, our work uses half of the training
iterations to reach an average test PSNR of >30.

本研究提出了一种方法，利用目标场景的几何先验来加速 3D 场景和物体的重建，以实现在移动设备上的即时重建。