We introduce MultiDiff, a novel approach for consistent novel view synthesis
of scenes from a single RGB image. The task of synthesizing novel views from a
single reference image is highly ill-posed by nature, as there exist multiple,
plausible explanations for unobserved areas. To address this issue, we
incorporate strong priors in form of monocular depth predictors and
video-diffusion models. Monocular depth enables us to condition our model on
warped reference images for the target views, increasing geometric stability.
The video-diffusion prior provides a strong proxy for 3D scenes, allowing the
model to learn continuous and pixel-accurate correspondences across generated
images. In contrast to approaches relying on autoregressive image generation
that are prone to drifts and error accumulation, MultiDiff jointly synthesizes
a sequence of frames yielding high-quality and multi-view consistent results --
even for long-term scene generation with large camera movements, while reducing
inference time by an order of magnitude. For additional consistency and image
quality improvements, we introduce a novel, structured noise distribution. Our
experimental results demonstrate that MultiDiff outperforms state-of-the-art
methods on the challenging, real-world datasets RealEstate10K and ScanNet.
Finally, our model naturally supports multi-view consistent editing without the
need for further tuning.

提出了 MultiDiff 方法，它是一种用于从单个 RGB 图像一致地合成新视角的新方法，通过引入单目深度预测器和视频扩散模型作为强先验条件，实现了高质量和多视角一致的结果。

MultiDiff: 从单幅图像实现一致的新视角合成

MultiDiff: Consistent Novel View Synthesis from a Single Image

We propose GaussCtrl, a text-driven method to edit a 3D scene reconstructed
by the 3D Gaussian Splatting (3DGS).
Our method first renders a collection of images by using the 3DGS and edits
them by using a pre-trained 2D diffusion model (ControlNet) based on the input
prompt, which is then used to optimise the 3D model.
Our key contribution is multi-view consistent editing, which enables editing
all images together instead of iteratively editing one image while updating the
3D model as in previous works.
It leads to faster editing as well as higher visual quality.
This is achieved by the two terms:
(a) depth-conditioned editing that enforces geometric consistency across
multi-view images by leveraging naturally consistent depth maps.
(b) attention-based latent code alignment that unifies the appearance of
edited images by conditioning their editing to several reference views through
self and cross-view attention between images' latent representations.
Experiments demonstrate that our method achieves faster editing and better
visual results than previous state-of-the-art methods.

提出了 GaussCtrl，一种使用文本驱动的方法来编辑由 3D Gaussian Splatting (3DGS) 重建的 3D 场景。该方法首先通过使用 3DGS 渲染一系列图像，然后使用基于输入提示的预训练二维扩散模型 (ControlNet) 对这些图像进行编辑，从而优化 3D 模型。我们的主要贡献是多视角一致性编辑，使得能够同时编辑所有图像，而不像以前的方法那样需要逐个编辑图像并更新 3D 模型。这导致更快的编辑速度和更高的视觉质量。这是通过以下两个方面实现的：(a) 基于深度的编辑，通过利用天然一致的深度图加强了多视角图像之间的几何一致性。(b) 基于注意力的潜在代码对齐，通过在图像的潜在表示之间进行自注意力和跨视图注意力，使编辑图像的外观统一于多个参考视图。实验表明，与之前的最先进方法相比，我们的方法实现了更快的编辑速度和更好的视觉效果。

GaussCtrl: 多视图一致的文本驱动 3D 高斯点绘制编辑

GaussCtrl: Multi-View Consistent Text-Driven 3D Gaussian Splatting  Editing

With the recent advances in NeRF-based 3D aware GANs quality, projecting an
image into the latent space of these 3D-aware GANs has a natural advantage over
2D GAN inversion: not only does it allow multi-view consistent editing of the
projected image, but it also enables 3D reconstruction and novel view synthesis
when given only a single image. However, the explicit viewpoint control acts as
a main hindrance in the 3D GAN inversion process, as both camera pose and
latent code have to be optimized simultaneously to reconstruct the given image.
Most works that explore the latent space of the 3D-aware GANs rely on
ground-truth camera viewpoint or deformable 3D model, thus limiting their
applicability. In this work, we introduce a generalizable 3D GAN inversion
method that infers camera viewpoint and latent code simultaneously to enable
multi-view consistent semantic image editing. The key to our approach is to
leverage pre-trained estimators for better initialization and utilize the
pixel-wise depth calculated from NeRF parameters to better reconstruct the
given image. We conduct extensive experiments on image reconstruction and
editing both quantitatively and qualitatively, and further compare our results
with 2D GAN-based editing to demonstrate the advantages of utilizing the latent
space of 3D GANs. Additional results and visualizations are available at
this https URL .

本研究提出了一种通用的 3D GAN 反演方法，可以同时计算相机视角和潜变量代码，从而实现多视角一致的语义图像编辑。其关键在于利用预训练的估算器用于更好的初始化，并利用从 NeRF 参数计算的像素深度来更好地重构给定的图像。