With the development of diffusion-based customization methods like
DreamBooth, individuals now have access to train the models that can generate
their personalized images. Despite the convenience, malicious users have
misused these techniques to create fake images, thereby triggering a privacy
security crisis. In light of this, proactive adversarial attacks are proposed
to protect users against customization. The adversarial examples are trained to
distort the customization model's outputs and thus block the misuse. In this
paper, we propose DisDiff (Disrupting Diffusion), a novel adversarial attack
method to disrupt the diffusion model outputs. We first delve into the
intrinsic image-text relationships, well-known as cross-attention, and
empirically find that the subject-identifier token plays an important role in
guiding image generation. Thus, we propose the Cross-Attention Erasure module
to explicitly "erase" the indicated attention maps and disrupt the text
guidance. Besides,we analyze the influence of the sampling process of the
diffusion model on Projected Gradient Descent (PGD) attack and introduce a
novel Merit Sampling Scheduler to adaptively modulate the perturbation updating
amplitude in a step-aware manner. Our DisDiff outperforms the state-of-the-art
methods by 12.75% of FDFR scores and 7.25% of ISM scores across two facial
benchmarks and two commonly used prompts on average.

本文提出了 DisDiff（Disrupting Diffusion），一种破坏扩散模型输出的新型对抗攻击方法。通过运用 Cross-Attention Erasure 模块来显式 “擦除” 指示的注意力图，并分析扩散模型的采样过程对 PGD 攻击的影响，引入了一种新颖的 Merit Sampling Scheduler 来自适应地调节扰动更新振幅。在两个面部基准和两个常用的提示场景上，我们的 DisDiff 方法在 FDFR 分数上优于现有方法 12.75％，在 ISM 分数上优于现有方法 7.25％。