Despite the substantial advancements in Vision-Language Pre-training (VLP)
models, their susceptibility to adversarial attacks poses a significant
challenge. Existing work rarely studies the transferability of attacks on VLP
models, resulting in a substantial performance gap from white-box attacks. We
observe that prior work overlooks the interaction mechanisms between
modalities, which plays a crucial role in understanding the intricacies of VLP
models. In response, we propose a novel attack, called Collaborative Multimodal
Interaction Attack (CMI-Attack), leveraging modality interaction through
embedding guidance and interaction enhancement. Specifically, attacking text at
the embedding level while preserving semantics, as well as utilizing
interaction image gradients to enhance constraints on perturbations of texts
and images. Significantly, in the image-text retrieval task on Flickr30K
dataset, CMI-Attack raises the transfer success rates from ALBEF to TCL,
$\text{CLIP}_{\text{ViT}}$ and $\text{CLIP}_{\text{CNN}}$ by 8.11%-16.75% over
state-of-the-art methods. Moreover, CMI-Attack also demonstrates superior
performance in cross-task generalization scenarios. Our work addresses the
underexplored realm of transfer attacks on VLP models, shedding light on the
importance of modality interaction for enhanced adversarial robustness.

通过利用模态交互机制，我们提出了一种名为 CMI-Attack 的新型攻击方法，在保持语义不变的同时，利用嵌入指导和交互增强攻击文本的嵌入层，并利用交互图像梯度增强对文本和图像的扰动约束。在 Flickr30K 数据集的图像 - 文本检索任务中，CMI-Attack 相对于现有方法提高了 ALBEF、TCL、CLIP_ViT 和 CLIP_CNN 的转移成功率 8.11%-16.75%。此外，CMI-Attack 在跨任务泛化情景中也表现出卓越性能，填补了 Vision-Language 预训练模型转移攻击领域的研究空白，揭示了模态交互对增强对抗鲁棒性的重要性。