Understanding the vulnerability of large-scale pre-trained vision-language
models like CLIP against adversarial attacks is key to ensuring zero-shot
generalization capacity on various downstream tasks. State-of-the-art defense
mechanisms generally adopt prompt learning strategies for adversarial
fine-tuning to improve the adversarial robustness of the pre-trained model
while keeping the efficiency of adapting to downstream tasks. Such a setup
leads to the problem of over-fitting which impedes further improvement of the
model's generalization capacity on both clean and adversarial examples. In this
work, we propose an adaptive Consistency-guided Adversarial Prompt Tuning
(i.e., CAPT) framework that utilizes multi-modal prompt learning to enhance the
alignment of image and text features for adversarial examples and leverage the
strong generalization of pre-trained CLIP to guide the model-enhancing its
robust generalization on adversarial examples while maintaining its accuracy on
clean ones. We also design a novel adaptive consistency objective function to
balance the consistency of adversarial inputs and clean inputs between the
fine-tuning model and the pre-trained model. We conduct extensive experiments
across 14 datasets and 4 data sparsity schemes (from 1-shot to full training
data settings) to show the superiority of CAPT over other state-of-the-art
adaption methods. CAPT demonstrated excellent performance in terms of the
in-distribution performance and the generalization under input distribution
shift and across datasets.

通过多模态提示学习来提高图像和文本特征的对齐度，利用预训练的 CLIP 强大的泛化能力，引导模型在对抗性示例上增强鲁棒泛化能力，同时在干净示例上保持准确性。