We introduce MultiDiff, a novel approach for consistent novel view synthesis
of scenes from a single RGB image. The task of synthesizing novel views from a
single reference image is highly ill-posed by nature, as there exist multiple,
plausible explanations for unobserved areas. To address this issue, we
incorporate strong priors in form of monocular depth predictors and
video-diffusion models. Monocular depth enables us to condition our model on
warped reference images for the target views, increasing geometric stability.
The video-diffusion prior provides a strong proxy for 3D scenes, allowing the
model to learn continuous and pixel-accurate correspondences across generated
images. In contrast to approaches relying on autoregressive image generation
that are prone to drifts and error accumulation, MultiDiff jointly synthesizes
a sequence of frames yielding high-quality and multi-view consistent results --
even for long-term scene generation with large camera movements, while reducing
inference time by an order of magnitude. For additional consistency and image
quality improvements, we introduce a novel, structured noise distribution. Our
experimental results demonstrate that MultiDiff outperforms state-of-the-art
methods on the challenging, real-world datasets RealEstate10K and ScanNet.
Finally, our model naturally supports multi-view consistent editing without the
need for further tuning.

提出了 MultiDiff 方法，它是一种用于从单个 RGB 图像一致地合成新视角的新方法，通过引入单目深度预测器和视频扩散模型作为强先验条件，实现了高质量和多视角一致的结果。

MultiDiff: 从单幅图像实现一致的新视角合成

MultiDiff: Consistent Novel View Synthesis from a Single Image

Monocular depth predictors are typically trained on large-scale training sets
which are naturally biased w.r.t the distribution of camera poses. As a result,
trained predictors fail to make reliable depth predictions for testing examples
captured under uncommon camera poses. To address this issue, we propose two
novel techniques that exploit the camera pose during training and prediction.
First, we introduce a simple perspective-aware data augmentation that
synthesizes new training examples with more diverse views by perturbing the
existing ones in a geometrically consistent manner. Second, we propose a
conditional model that exploits the per-image camera pose as prior knowledge by
encoding it as a part of the input. We show that jointly applying the two
methods improves depth prediction on images captured under uncommon and even
never-before-seen camera poses. We show that our methods improve performance
when applied to a range of different predictor architectures. Lastly, we show
that explicitly encoding the camera pose distribution improves the
generalization performance of a synthetically trained depth predictor when
evaluated on real images.

提出两种新技术解决单目深度预测模型因训练数据相机位置偏差带来的泛化性能下降问题，一种基于数据增强的方法，一种基于条件模型的方法，实验证明两种方法能够显著提高模型在拍摄位置不同的图像中的预测效果和泛化性能。