We study text-based image editing (TBIE) of a single image by counterfactual
inference because it is an elegant formulation to precisely address the
requirement: the edited image should retain the fidelity of the original one.
Through the lens of the formulation, we find that the crux of TBIE is that
existing techniques hardly achieve a good trade-off between editability and
fidelity, mainly due to the overfitting of the single-image fine-tuning. To
this end, we propose a Doubly Abductive Counterfactual inference framework
(DAC). We first parameterize an exogenous variable as a UNet LoRA, whose
abduction can encode all the image details. Second, we abduct another exogenous
variable parameterized by a text encoder LoRA, which recovers the lost
editability caused by the overfitted first abduction. Thanks to the second
abduction, which exclusively encodes the visual transition from post-edit to
pre-edit, its inversion -- subtracting the LoRA -- effectively reverts pre-edit
back to post-edit, thereby accomplishing the edit. Through extensive
experiments, our DAC achieves a good trade-off between editability and
fidelity. Thus, we can support a wide spectrum of user editing intents,
including addition, removal, manipulation, replacement, style transfer, and
facial change, which are extensively validated in both qualitative and
quantitative evaluations. Codes are in this https URL

我们通过反事实推理的单一图像的基于文本的图像编辑（TBIE）进行研究，因为它是一个精确解决要求的优雅表述：编辑后的图像应保留原始图像的保真度。通过这个表述，我们发现 TBIE 的症结在于现有技术很难在可编辑性和保真度之间取得良好的折衷，主要是由于单一图像微调过拟合造成的。为了解决这个问题，我们提出了一个双重绑定反事实推理框架（DAC）。我们首先将一个外生变量参数化为 UNet LoRA，其绑定可以编码所有图像细节。其次，我们绑定另一个由文本编码器 LoRA 参数化的外生变量，它恢复了由过拟合的第一个绑定引起的失去的可编辑性。由于第二个绑定，它仅编码从编辑后到编辑前的视觉转换，其逆操作 - 减去 LoRA - 有效地将编辑前的图像恢复到编辑后的状态，从而实现了编辑。通过大量实验证明，我们的 DAC 在可编辑性和保真度之间取得了良好的折衷。因此，我们可以支持广泛的用户编辑意图，包括添加、删除、操作、替换、风格转换和面部变化，在定性和定量评估中得到了广泛验证。

基于文本的图像编辑的双重可推理对策

Doubly Abductive Counterfactual Inference for Text-based Image Editing

Text-based image editing has advanced significantly in recent years. With the
rise of diffusion models, image editing via textual instructions has become
ubiquitous. Unfortunately, current models lack the ability to customize the
quantity of the change per pixel or per image fragment, resorting to changing
the entire image in an equal amount, or editing a specific region using a
binary mask. In this paper, we suggest a new framework which enables the user
to customize the quantity of change for each image fragment, thereby enhancing
the flexibility and verbosity of modern diffusion models. Our framework does
not require model training or fine-tuning, but instead performs everything at
inference time, making it easily applicable to an existing model. We show both
qualitatively and quantitatively that our method allows better controllability
and can produce results which are unattainable by existing models. Our code is
available at: this https URL

提出一种新的框架，使用户能够为每个图像碎片自定义更改量，增强了现代扩散模型的灵活性和表达能力，且无需模型训练或微调，可直接应用于现有模型，量化和定性结果表明，该方法实现了更好的可控性，并可产生现有模型无法达到的结果。

差分扩散：赋予每个像素以其强度

Differential Diffusion: Giving Each Pixel Its Strength

We introduce Delta Denoising Score (DDS), a novel scoring function for
text-based image editing that guides minimal modifications of an input image
towards the content described in a target prompt. DDS leverages the rich
generative prior of text-to-image diffusion models and can be used as a loss
term in an optimization problem to steer an image towards a desired direction
dictated by a text. DDS utilizes the Score Distillation Sampling (SDS)
mechanism for the purpose of image editing. We show that using only SDS often
produces non-detailed and blurry outputs due to noisy gradients. To address
this issue, DDS uses a prompt that matches the input image to identify and
remove undesired erroneous directions of SDS. Our key premise is that SDS
should be zero when calculated on pairs of matched prompts and images, meaning
that if the score is non-zero, its gradients can be attributed to the erroneous
component of SDS. Our analysis demonstrates the competence of DDS for text
based image-to-image translation. We further show that DDS can be used to train
an effective zero-shot image translation model. Experimental results indicate
that DDS outperforms existing methods in terms of stability and quality,
highlighting its potential for real-world applications in text-based image
editing.

Delta Denoising Score 是一种针对基于文本的图像编辑的新的评分函数，使用在最小修改输入图像为所需内容的过程中，利用文本生成模型的生成先验，通过 SDS 机制作为图像编辑的损失项，使用 DDS 去识别和去除 SDS 的误差方向，从而在文本的指导下实现图像到图像的翻译和零样本的图像翻译。