Guided image synthesis methods, like SDEdit based on the diffusion model,
excel at creating realistic images from user inputs such as stroke paintings.
However, existing efforts mainly focus on image quality, often overlooking a
key point: the diffusion model represents a data distribution, not individual
images. This introduces a low but critical chance of generating images that
contradict user intentions, raising ethical concerns. For example, a user
inputting a stroke painting with female characteristics might, with some
probability, get male faces from SDEdit. To expose this potential
vulnerability, we aim to build an adversarial attack forcing SDEdit to generate
a specific data distribution aligned with a specified attribute (e.g., female),
without changing the input's attribute characteristics. We propose the Targeted
Attribute Generative Attack (TAGA), using an attribute-aware objective function
and optimizing the adversarial noise added to the input stroke painting.
Empirical studies reveal that traditional adversarial noise struggles with
TAGA, while natural perturbations like exposure and motion blur easily alter
generated images' attributes. To execute effective attacks, we introduce
FoolSDEdit: We design a joint adversarial exposure and blur attack, adding
exposure and motion blur to the stroke painting and optimizing them together.
We optimize the execution strategy of various perturbations, framing it as a
network architecture search problem. We create the SuperPert, a graph
representing diverse execution strategies for different perturbations. After
training, we obtain the optimized execution strategy for effective TAGA against
SDEdit. Comprehensive experiments on two datasets show our method compelling
SDEdit to generate a targeted attribute-aware data distribution, significantly
outperforming baselines.

通过引入对抗噪声来执行有针对性的属性生成对 SDEdit 的有效攻击，同时利用属性感知型目标函数和优化对输入笔触绘画添加的对抗噪声。

FoolSDEdit：将您的编辑巧妙地引导至目标属性感知分布

FoolSDEdit: Deceptively Steering Your Edits Towards Targeted  Attribute-aware Distribution

Controllable image synthesis with user scribbles has gained huge public
interest with the recent advent of text-conditioned latent diffusion models.
The user scribbles control the color composition while the text prompt provides
control over the overall image semantics. However, we note that prior works in
this direction suffer from an intrinsic domain shift problem, wherein the
generated outputs often lack details and resemble simplistic representations of
the target domain. In this paper, we propose a novel guided image synthesis
framework, which addresses this problem by modeling the output image as the
solution of a constrained optimization problem. We show that while computing an
exact solution to the optimization is infeasible, an approximation of the same
can be achieved while just requiring a single pass of the reverse diffusion
process. Additionally, we show that by simply defining a cross-attention based
correspondence between the input text tokens and the user stroke-painting, the
user is also able to control the semantics of different painted regions without
requiring any conditional training or finetuning. Human user study results show
that the proposed approach outperforms the previous state-of-the-art by over
85.32% on the overall user satisfaction scores. Project page for our paper is
available at this https URL.

本文提出了一种新的引导图像合成框架，该框架通过将输出图像建模为受约束优化问题的解决方案来解决领域偏移问题。同时，本文还展示了通过定义基于交叉注意力的输入文本符号和用户笔画之间的对应关系，用户可以在不需要任何条件训练或微调的情况下控制不同绘制区域的语义。