Diffusion models have demonstrated impressive performance in text-guided
image generation. To leverage the knowledge of text-guided image generation
models in image editing, current approaches either fine-tune the pretrained
models using the input image (e.g., Imagic) or incorporate structure
information as additional constraints into the pretrained models (e.g.,
ControlNet). However, fine-tuning large-scale diffusion models on a single
image can lead to severe overfitting issues and lengthy inference time. The
information leakage from pretrained models makes it challenging to preserve the
text-irrelevant content of the input image while generating new features guided
by language descriptions. On the other hand, methods that incorporate
structural guidance (e.g., edge maps, semantic maps, keypoints) as additional
constraints face limitations in preserving other attributes of the original
image, such as colors or textures. A straightforward way to incorporate the
original image is to directly use it as an additional control. However, since
image editing methods are typically trained on the image reconstruction task,
the incorporation can lead to the identical mapping issue, where the model
learns to output an image identical to the input, resulting in limited editing
capabilities. To address these challenges, we propose a text-to-image editing
model with Image Information Removal module (IIR) to selectively erase
color-related and texture-related information from the original image, allowing
us to better preserve the text-irrelevant content and avoid the identical
mapping issue. We evaluate our model on three benchmark datasets: CUB, Outdoor
Scenes, and COCO. Our approach achieves the best editability-fidelity
trade-off, and our edited images are approximately 35% more preferred by
annotators than the prior-arts on COCO.

本文提出了一种利用 Image Information Removal 模块来精准地去除原始图像中与颜色和纹理相关的信息，以更好地保存文本不相关内容并避免相同映射问题，从而实现文本到图像编辑的方法。我们的方法在三个基准数据集上达到了最佳的可编辑性 - 保真度的平衡，并且我们的编辑图像比 COCO 上的之前的艺术作品更受注释者的欢迎。