Targeted transfer-based attacks involving adversarial examples pose a
significant threat to large visual-language models (VLMs). However, the
state-of-the-art (SOTA) transfer-based attacks incur high costs due to
excessive iteration counts. Furthermore, the generated adversarial examples
exhibit pronounced adversarial noise and demonstrate limited efficacy in
evading defense methods such as DiffPure. To address these issues, inspired by
score matching, we introduce AdvDiffVLM, which utilizes diffusion models to
generate natural, unrestricted adversarial examples. Specifically, AdvDiffVLM
employs Adaptive Ensemble Gradient Estimation to modify the score during the
diffusion model's reverse generation process, ensuring the adversarial examples
produced contain natural adversarial semantics and thus possess enhanced
transferability. Simultaneously, to enhance the quality of adversarial examples
further, we employ the GradCAM-guided Mask method to disperse adversarial
semantics throughout the image, rather than concentrating them in a specific
area. Experimental results demonstrate that our method achieves a speedup
ranging from 10X to 30X compared to existing transfer-based attack methods,
while maintaining superior quality of adversarial examples. Additionally, the
generated adversarial examples possess strong transferability and exhibit
increased robustness against adversarial defense methods. Notably, AdvDiffVLM
can successfully attack commercial VLMs, including GPT-4V, in a black-box
manner.

传统的迁移攻击成本高，噪音明显且对防御方法难以有效回避。本文通过生成自然、非受限的对抗样本，提出了 AdvDiffVLM 方法，利用扩散模型和自适应集成梯度估计改善了传统方法的缺陷，并通过 GradCAM-guided Mask 方法提高了样本质量。实验结果显示，我们的方法在速度上比现有方法快 10 倍至 30 倍，并保持了超强的对抗样本质量。此外，生成的对抗样本在攻击上具有强大的迁移性，并对对抗性防御方法表现出增强的抗性。值得注意的是，AdvDiffVLM 可以以黑盒方式成功攻击商业化的大型视觉语言模型，包括 GPT-4V。