In this paper, we focus on the One-shot Novel View Synthesis (O-NVS) task
which targets synthesizing photo-realistic novel views given only one reference
image per scene. Previous One-shot Generalizable Neural Radiance Fields
(OG-NeRF) methods solve this task in an inference-time finetuning-free manner,
yet suffer the blurry issue due to the encoder-only architecture that highly
relies on the limited reference image. On the other hand, recent
diffusion-based image-to-3d methods show vivid plausible results via distilling
pre-trained 2D diffusion models into a 3D representation, yet require tedious
per-scene optimization. Targeting these issues, we propose the GD^2-NeRF, a
Generative Detail compensation framework via GAN and Diffusion that is both
inference-time finetuning-free and with vivid plausible details. In detail,
following a coarse-to-fine strategy, GD^2-NeRF is mainly composed of a
One-stage Parallel Pipeline (OPP) and a 3D-consistent Detail Enhancer
(Diff3DE). At the coarse stage, OPP first efficiently inserts the GAN model
into the existing OG-NeRF pipeline for primarily relieving the blurry issue
with in-distribution priors captured from the training dataset, achieving a
good balance between sharpness (LPIPS, FID) and fidelity (PSNR, SSIM). Then, at
the fine stage, Diff3DE further leverages the pre-trained image diffusion
models to complement rich out-distribution details while maintaining decent 3D
consistency. Extensive experiments on both the synthetic and real-world
datasets show that GD$^2$-NeRF noticeably improves the details while without
per-scene finetuning.

本研究着重于一次全新视图合成（O-NVS）任务，目标是在每个场景只给定一张参考图像的情况下合成逼真的新视图。提出了一种使用 GAN 和扩散的生成性细节补偿框架 GD^2-NeRF，该框架在推理时不需要微调，并且具有生动逼真的细节。

基于 GAN 和扩散的 GD^2-NeRF：一次性通用的神经辐射场生成细节补偿

GD^2-NeRF: Generative Detail Compensation via GAN and Diffusion for  One-shot Generalizable Neural Radiance Fields

Text-guided diffusion models have shown superior performance in image/video
generation and editing. While few explorations have been performed in 3D
scenarios. In this paper, we discuss three fundamental and interesting problems
on this topic. First, we equip text-guided diffusion models to achieve
$\textbf{3D-consistent generation}$. Specifically, we integrate a NeRF-like
neural field to generate low-resolution coarse results for a given camera view.
Such results can provide 3D priors as condition information for the following
diffusion process. During denoising diffusion, we further enhance the 3D
consistency by modeling cross-view correspondences with a novel two-stream
(corresponding to two different views) asynchronous diffusion process. Second,
we study $\textbf{3D local editing}$ and propose a two-step solution that can
generate 360$^{\circ}$ manipulated results by editing an object from a single
view. Step 1, we propose to perform 2D local editing by blending the predicted
noises. Step 2, we conduct a noise-to-text inversion process that maps 2D
blended noises into the view-independent text embedding space. Once the
corresponding text embedding is obtained, 360$^{\circ}$ images can be
generated. Last but not least, we extend our model to perform \textbf{one-shot
novel view synthesis} by fine-tuning on a single image, firstly showing the
potential of leveraging text guidance for novel view synthesis. Extensive
experiments and various applications show the prowess of our 3DDesigner. The
project page is available at this https URL

本文提出了一种基于文本引导扩散模型的 3D 场景生成、编辑和新视角合成方法，并重点讨论了 3D 一致性、本地编辑和单张图像训练等基础问题，取得了较好的效果。