Pretrained vision-language models (VLMs) like CLIP have shown impressive generalization performance across various downstream tasks, yet they remain vulnerable to adversarial attacks. While prior research has primarily concentrated on improving the adversarial robustness of image encoders to guard against attacks on images, the exploration of text-based and multimodal attacks has largely been overlooked. In this work, we initiate the first known and comprehensive effort to study adapting vision-language models for adversarial robustness under the multimodal attack. Firstly, we introduce a multimodal attack strategy and investigate the impact of different attacks. We then propose a multimodal contrastive adversarial training loss, aligning the clean and adversarial text embeddings with the adversarial and clean visual features, to enhance the adversarial robustness of both image and text encoders of CLIP. Extensive experiments on 15 datasets across two tasks demonstrate that our method significantly improves the adversarial robustness of CLIP. Interestingly, we find that the model fine-tuned against multimodal adversarial attacks exhibits greater robustness than its counterpart fine-tuned solely against image-based attacks, even in the context of image attacks, which may open up new possibilities for enhancing the security of VLMs.

通过研究视觉语言模型（VLMs）的自适应对抗性鲁棒性，我们引入了一种多模态对抗性攻击策略，并在图像和文本编码器上采用多模态对比对抗性训练损失，以提高CLIP的对抗性鲁棒性。在15个数据集上进行的广泛实验表明，我们的方法显著提高了CLIP的对抗性鲁棒性，甚至在图像攻击的背景下，经过多模态对抗性攻击的模型表现出比仅进行图像攻击微调的模型更高的鲁棒性。

再审视视觉语言模型的对抗鲁棒性：一种多模态的观点