Text-guided diffusion models have shown superior performance in image/video
generation and editing. While few explorations have been performed in 3D
scenarios. In this paper, we discuss three fundamental and interesting problems
on this topic. First, we equip text-guided diffusion models to achieve
$\textbf{3D-consistent generation}$. Specifically, we integrate a NeRF-like
neural field to generate low-resolution coarse results for a given camera view.
Such results can provide 3D priors as condition information for the following
diffusion process. During denoising diffusion, we further enhance the 3D
consistency by modeling cross-view correspondences with a novel two-stream
(corresponding to two different views) asynchronous diffusion process. Second,
we study $\textbf{3D local editing}$ and propose a two-step solution that can
generate 360$^{\circ}$ manipulated results by editing an object from a single
view. Step 1, we propose to perform 2D local editing by blending the predicted
noises. Step 2, we conduct a noise-to-text inversion process that maps 2D
blended noises into the view-independent text embedding space. Once the
corresponding text embedding is obtained, 360$^{\circ}$ images can be
generated. Last but not least, we extend our model to perform \textbf{one-shot
novel view synthesis} by fine-tuning on a single image, firstly showing the
potential of leveraging text guidance for novel view synthesis. Extensive
experiments and various applications show the prowess of our 3DDesigner. The
project page is available at this https URL

本文提出了一种基于文本引导扩散模型的 3D 场景生成、编辑和新视角合成方法，并重点讨论了 3D 一致性、本地编辑和单张图像训练等基础问题，取得了较好的效果。