This paper proposes Instruct 4D-to-4D that achieves 4D awareness and
spatial-temporal consistency for 2D diffusion models to generate high-quality
instruction-guided dynamic scene editing results. Traditional applications of
2D diffusion models in dynamic scene editing often result in inconsistency,
primarily due to their inherent frame-by-frame editing methodology. Addressing
the complexities of extending instruction-guided editing to 4D, our key insight
is to treat a 4D scene as a pseudo-3D scene, decoupled into two sub-problems:
achieving temporal consistency in video editing and applying these edits to the
pseudo-3D scene. Following this, we first enhance the Instruct-Pix2Pix (IP2P)
model with an anchor-aware attention module for batch processing and consistent
editing. Additionally, we integrate optical flow-guided appearance propagation
in a sliding window fashion for more precise frame-to-frame editing and
incorporate depth-based projection to manage the extensive data of pseudo-3D
scenes, followed by iterative editing to achieve convergence. We extensively
evaluate our approach in various scenes and editing instructions, and
demonstrate that it achieves spatially and temporally consistent editing
results, with significantly enhanced detail and sharpness over the prior art.
Notably, Instruct 4D-to-4D is general and applicable to both monocular and
challenging multi-camera scenes. Code and more results are available at
immortalco.github.io/Instruct-4D-to-4D.

本文提出了 Instruct 4D-to-4D 方法，用于实现 2D 扩散模型的 4D 感知和时空一致性，以生成高质量的指导动态场景编辑结果。

4D 场景的 4D 编辑：利用 2D 扩散将其编辑为伪 3D 场景

Instruct 4D-to-4D: Editing 4D Scenes as Pseudo-3D Scenes Using 2D  Diffusion

Recent years have witnessed considerable achievements in editing images with
text instructions. When applying these editors to dynamic scene editing, the
new-style scene tends to be temporally inconsistent due to the frame-by-frame
nature of these 2D editors. To tackle this issue, we propose Control4D, a novel
approach for high-fidelity and temporally consistent 4D portrait editing.
Control4D is built upon an efficient 4D representation with a 2D
diffusion-based editor. Instead of using direct supervisions from the editor,
our method learns a 4D GAN from it and avoids the inconsistent supervision
signals. Specifically, we employ a discriminator to learn the generation
distribution based on the edited images and then update the generator with the
discrimination signals. For more stable training, multi-level information is
extracted from the edited images and used to facilitate the learning of the
generator. Experimental results show that Control4D surpasses previous
approaches and achieves more photo-realistic and consistent 4D editing
performances. The link to our project website is
this https URL

本研究提出了一种基于 4D 表示和 2D 扩散编辑器的 Control4D 方法，通过使用肖像编辑生成的鉴别器以改进发生式 4D 肖像编辑的稳定性。