Recent research has demonstrated that the combination of pretrained diffusion
models with neural radiance fields (NeRFs) has emerged as a promising approach
for text-to-3D generation. Simply coupling NeRF with diffusion models will
result in cross-view inconsistency and degradation of stylized view syntheses.
To address this challenge, we propose the Edit-DiffNeRF framework, which is
composed of a frozen diffusion model, a proposed delta module to edit the
latent semantic space of the diffusion model, and a NeRF. Instead of training
the entire diffusion for each scene, our method focuses on editing the latent
semantic space in frozen pretrained diffusion models by the delta module. This
fundamental change to the standard diffusion framework enables us to make
fine-grained modifications to the rendered views and effectively consolidate
these instructions in a 3D scene via NeRF training. As a result, we are able to
produce an edited 3D scene that faithfully aligns to input text instructions.
Furthermore, to ensure semantic consistency across different viewpoints, we
propose a novel multi-view semantic consistency loss that extracts a latent
semantic embedding from the input view as a prior, and aim to reconstruct it in
different views. Our proposed method has been shown to effectively edit
real-world 3D scenes, resulting in 25% improvement in the alignment of the
performed 3D edits with text instructions compared to prior work.

本文提出了一种名为 Edit-DiffNeRF 的框架，通过对已经训练好的扩散模型的潜在语义空间进行微调以有效地合并这些指令，从而实现对 3D 场景的微调，同时通过多视图语义一致性损失确保在不同的视图中重建潜在的语义嵌入。