Novel-view synthesis through diffusion models has demonstrated remarkable
potential for generating diverse and high-quality images. Yet, the independent
process of image generation in these prevailing methods leads to challenges in
maintaining multiple-view consistency. To address this, we introduce
ViewFusion, a novel, training-free algorithm that can be seamlessly integrated
into existing pre-trained diffusion models. Our approach adopts an
auto-regressive method that implicitly leverages previously generated views as
context for the next view generation, ensuring robust multi-view consistency
during the novel-view generation process. Through a diffusion process that
fuses known-view information via interpolated denoising, our framework
successfully extends single-view conditioned models to work in multiple-view
conditional settings without any additional fine-tuning. Extensive experimental
results demonstrate the effectiveness of ViewFusion in generating consistent
and detailed novel views.

通过扩散模型进行新视图合成表现出了出色的潜力，然而这些常见方法中图像生成的独立过程导致在保持多视图一致性方面存在挑战。为了解决这个问题，我们引入了 ViewFusion，这是一种新颖的、无需训练的算法，可以无缝地集成到现有的预先训练好的扩散模型中。我们的方法采用自回归的方式，隐含地利用之前生成的视图作为下一个视图生成过程的上下文，确保在新视图生成过程中具有稳健的多视图一致性。通过通过插值去噪将已知视图信息融合到扩散过程中，我们的框架成功地将单视图条件模型扩展为能在多视图条件设置下工作，无需额外的微调。广泛的实验结果表明 ViewFusion 在生成一致且详细的新视图方面的有效性。

多视角一致性插值去噪视图融合

ViewFusion: Towards Multi-View Consistency via Interpolated Denoising

Deep learning is providing a wealth of new approaches to the old problem of
novel view synthesis, from Neural Radiance Field (NeRF) based approaches to
end-to-end style architectures. Each approach offers specific strengths but
also comes with specific limitations in their applicability. This work
introduces ViewFusion, a state-of-the-art end-to-end generative approach to
novel view synthesis with unparalleled flexibility. ViewFusion consists in
simultaneously applying a diffusion denoising step to any number of input views
of a scene, then combining the noise gradients obtained for each view with an
(inferred) pixel-weighting mask, ensuring that for each region of the target
scene only the most informative input views are taken into account. Our
approach resolves several limitations of previous approaches by (1) being
trainable and generalizing across multiple scenes and object classes, (2)
adaptively taking in a variable number of pose-free views at both train and
test time, (3) generating plausible views even in severely undetermined
conditions (thanks to its generative nature) -- all while generating views of
quality on par or even better than state-of-the-art methods. Limitations
include not generating a 3D embedding of the scene, resulting in a relatively
slow inference speed, and our method only being tested on the relatively small
dataset NMR. Code is available.

通过深度学习，本研究提出了一种新颖的视角合成方法 ViewFusion，其以端到端的生成方式灵活地综合多个输入视角，消除噪声并生成高质量的视角，相较于现有方法具有更好的泛化性能和适用性。