This paper proposes ConsistDreamer - a novel framework that lifts 2D
diffusion models with 3D awareness and 3D consistency, thus enabling
high-fidelity instruction-guided scene editing. To overcome the fundamental
limitation of missing 3D consistency in 2D diffusion models, our key insight is
to introduce three synergetic strategies that augment the input of the 2D
diffusion model to become 3D-aware and to explicitly enforce 3D consistency
during the training process. Specifically, we design surrounding views as
context-rich input for the 2D diffusion model, and generate 3D-consistent,
structured noise instead of image-independent noise. Moreover, we introduce
self-supervised consistency-enforcing training within the per-scene editing
procedure. Extensive evaluation shows that our ConsistDreamer achieves
state-of-the-art performance for instruction-guided scene editing across
various scenes and editing instructions, particularly in complicated
large-scale indoor scenes from ScanNet++, with significantly improved sharpness
and fine-grained textures. Notably, ConsistDreamer stands as the first work
capable of successfully editing complex (e.g., plaid/checkered) patterns. Our
project page is at immortalco.github.io/ConsistDreamer.

提出了一种名为 ConsistDreamer 的新框架，该框架在 2D 扩散模型中引入了 3D 感知和 3D 一致性，从而实现了高保真度的指导场景编辑。

ConsistDreamer: 高保真场景编辑的三维一致的二维扩散

ConsistDreamer: 3D-Consistent 2D Diffusion for High-Fidelity Scene  Editing

Recent advances in large-scale pretraining have yielded visual foundation
models with strong capabilities. Not only can recent models generalize to
arbitrary images for their training task, their intermediate representations
are useful for other visual tasks such as detection and segmentation. Given
that such models can classify, delineate, and localize objects in 2D, we ask
whether they also represent their 3D structure? In this work, we analyze the 3D
awareness of visual foundation models. We posit that 3D awareness implies that
representations (1) encode the 3D structure of the scene and (2) consistently
represent the surface across views. We conduct a series of experiments using
task-specific probes and zero-shot inference procedures on frozen features. Our
experiments reveal several limitations of the current models. Our code and
analysis can be found at this https URL

最近大规模预训练的进步提供了具有强大功能的视觉基础模型。我们分析了视觉基础模型的三维感知能力，并通过一系列实验揭示了当前模型的几个局限性。