We propose Instruct2Attack (I2A), a language-guided semantic attack that
generates semantically meaningful perturbations according to free-form language
instructions. We make use of state-of-the-art latent diffusion models, where we
adversarially guide the reverse diffusion process to search for an adversarial
latent code conditioned on the input image and text instruction. Compared to
existing noise-based and semantic attacks, I2A generates more natural and
diverse adversarial examples while providing better controllability and
interpretability. We further automate the attack process with GPT-4 to generate
diverse image-specific text instructions. We show that I2A can successfully
break state-of-the-art deep neural networks even under strong adversarial
defenses, and demonstrate great transferability among a variety of network
architectures.

我们提出了 Instruct2Attack（I2A），一种基于语言引导的语义攻击，根据自由形式的语言指令生成语义上有意义的扰动。我们利用最先进的潜态扩散模型，通过对逆扩散过程进行对抗性引导，寻找输入图像和文本指令条件下的对抗性潜态编码。与基于噪声和语义的现有攻击相比，I2A 生成更自然和多样的对抗性样本，同时提供更好的可控性和可解释性。我们还利用 GPT-4 自动化攻击过程，生成多样的图像特定文本指令。我们表明，即使在强大的对抗性防御下，I2A 仍能成功破解最先进的深度神经网络，并展示其在各种网络架构之间的强大迁移能力。