Synthesizing novel view images from a few views is a challenging but
practical problem. Existing methods often struggle with producing high-quality
results or necessitate per-object optimization in such few-view settings due to
the insufficient information provided. In this work, we explore leveraging the
strong 2D priors in pre-trained diffusion models for synthesizing novel view
images. 2D diffusion models, nevertheless, lack 3D awareness, leading to
distorted image synthesis and compromising the identity. To address these
problems, we propose DreamSparse, a framework that enables the frozen
pre-trained diffusion model to generate geometry and identity-consistent novel
view image. Specifically, DreamSparse incorporates a geometry module designed
to capture 3D features from sparse views as a 3D prior. Subsequently, a spatial
guidance model is introduced to convert these 3D feature maps into spatial
information for the generative process. This information is then used to guide
the pre-trained diffusion model, enabling it to generate geometrically
consistent images without tuning it. Leveraging the strong image priors in the
pre-trained diffusion models, DreamSparse is capable of synthesizing
high-quality novel views for both object and scene-level images and
generalising to open-set images. Experimental results demonstrate that our
framework can effectively synthesize novel view images from sparse views and
outperforms baselines in both trained and open-set category images. More
results can be found on our project page:
this https URL

本研究中，我们探讨了如何在缺乏信息情况下，运用预先训练的扩散模型来合成全新视角的高质量图像，提出了 DreamSparse 框架，并利用几何模块来抓取 3D 特征，将其转化成空间信息来指导生成过程，并通过改进 2D 扩散模型的方式来保证其生成几何一致的图像，通过实验证明了该方法的有效性。