Vision-language models (VLMs) seamlessly integrate visual and textual data to
perform tasks such as image classification, caption generation, and visual
question answering. However, adversarial images often struggle to deceive all
prompts effectively in the context of cross-prompt migration attacks, as the
probability distribution of the tokens in these images tends to favor the
semantics of the original image rather than the target tokens. To address this
challenge, we propose a Contextual-Injection Attack (CIA) that employs
gradient-based perturbation to inject target tokens into both visual and
textual contexts, thereby improving the probability distribution of the target
tokens. By shifting the contextual semantics towards the target tokens instead
of the original image semantics, CIA enhances the cross-prompt transferability
of adversarial images.Extensive experiments on the BLIP2, InstructBLIP, and
LLaVA models show that CIA outperforms existing methods in cross-prompt
transferability, demonstrating its potential for more effective adversarial
strategies in VLMs.

通过使用梯度方法对视觉和文本上下文进行干扰，Contextual-Injection Attack（CIA）可以将目标标记注入视觉和文本环境，从而改善目标标记的概率分布，进而增强对抗图像的跨提示可迁移性，对于视觉 - 语言模型（VLMs）中的更有效的对抗策略具有潜力。