Pretrained vision-language models (VLMs) like CLIP have shown impressive
generalization performance across various downstream tasks, yet they remain
vulnerable to adversarial attacks. While prior research has primarily
concentrated on improving the adversarial robustness of image encoders to guard
against attacks on images, the exploration of text-based and multimodal attacks
has largely been overlooked. In this work, we initiate the first known and
comprehensive effort to study adapting vision-language models for adversarial
robustness under the multimodal attack. Firstly, we introduce a multimodal
attack strategy and investigate the impact of different attacks. We then
propose a multimodal contrastive adversarial training loss, aligning the clean
and adversarial text embeddings with the adversarial and clean visual features,
to enhance the adversarial robustness of both image and text encoders of CLIP.
Extensive experiments on 15 datasets across two tasks demonstrate that our
method significantly improves the adversarial robustness of CLIP.
Interestingly, we find that the model fine-tuned against multimodal adversarial
attacks exhibits greater robustness than its counterpart fine-tuned solely
against image-based attacks, even in the context of image attacks, which may
open up new possibilities for enhancing the security of VLMs.

通过研究视觉语言模型（VLMs）的自适应对抗性鲁棒性，我们引入了一种多模态对抗性攻击策略，并在图像和文本编码器上采用多模态对比对抗性训练损失，以提高 CLIP 的对抗性鲁棒性。在 15 个数据集上进行的广泛实验表明，我们的方法显著提高了 CLIP 的对抗性鲁棒性，甚至在图像攻击的背景下，经过多模态对抗性攻击的模型表现出比仅进行图像攻击微调的模型更高的鲁棒性。