Despite recent advancements in neural 3D reconstruction, the dependence on dense multi-view captures restricts their broader applicability. In this work, we propose \textbf{ViewCrafter}, a novel method for synthesizing high-fidelity novel views of generic scenes from single or sparse images with the prior of video diffusion model. Our method takes advantage of the powerful generation capabilities of video diffusion model and the coarse 3D clues offered by point-based representation to generate high-quality video frames with precise camera pose control. To further enlarge the generation range of novel views, we tailored an iterative view synthesis strategy together with a camera trajectory planning algorithm to progressively extend the 3D clues and the areas covered by the novel views. With ViewCrafter, we can facilitate various applications, such as immersive experiences with real-time rendering by efficiently optimizing a 3D-GS representation using the reconstructed 3D points and the generated novel views, and scene-level text-to-3D generation for more imaginative content creation. Extensive experiments on diverse datasets demonstrate the strong generalization capability and superior performance of our method in synthesizing high-fidelity and consistent novel views.

本研究解决了现有神经3D重建方法对密集多视图捕获的依赖性，从而限制了其广泛应用的问题。我们提出的ViewCrafter方法利用视频扩散模型的强大生成能力和基于点的表示提供的粗略3D线索，从单张或稀疏图像中合成高保真新视图。此外，我们的迭代视图合成策略和相机轨迹规划算法进一步扩展了新视图的生成范围。实验表明，ViewCrafter在生成高保真且一致的新视图方面具有强大的泛化能力和卓越的性能。

ViewCrafter：驯化视频扩散模型以实现高保真新视图合成