This work addresses the challenge of achieving zero-shot adversarial
robustness while preserving zero-shot generalization in large-scale foundation
models, with a focus on the popular Contrastive Language-Image Pre-training
(CLIP). Although foundation models were reported to have exceptional zero-shot
generalization, they are highly vulnerable to adversarial perturbations.
Existing methods achieve a comparable good tradeoff between zero-shot
adversarial robustness and generalization under small adversarial
perturbations. However, they fail to achieve a good tradeoff under large
adversarial perturbations. To this end, we propose a novel Text-Image Mutual
Awareness (TIMA) method that strikes a balance between zero-shot adversarial
robustness and generalization. More precisely, we propose an Image-Aware Text
(IAT) tuning mechanism that increases the inter-class distance of text
embeddings by incorporating the Minimum Hyperspherical Energy (MHE).
Simultaneously, fixed pre-trained image embeddings are used as cross-modal
auxiliary supervision to maintain the similarity between the MHE-tuned and
original text embeddings by the knowledge distillation, preserving semantic
information between different classes. Besides, we introduce a Text-Aware Image
(TAI) tuning mechanism, which increases inter-class distance between image
embeddings during the training stage by Text-distance based Adaptive Margin
(TAM). Similarly, a knowledge distillation is utilized to retain the similarity
between fine-tuned and pre-trained image embeddings. Extensive experimental
results demonstrate the effectiveness of our approach, showing impressive
zero-shot performance against a wide range of adversarial perturbations while
preserving the zero-shot generalization capabilities of the original CLIP
model.

通过引入一种新的文本图像相互感知（TIMA）方法，这项工作提出了一种旨在在大规模基础模型中实现零 - shot 对抗鲁棒性并保持零 - shot 泛化的挑战，重点关注流行的对比语言图像预训练（CLIP）模型。该方法在小型对抗扰动下在零 - shot 对抗鲁棒性和泛化之间取得了可比的良好权衡，但在大型对抗扰动下未能实现良好的权衡。该方法在文本嵌入之间增加了最小超球能量（MHE）以提高文本嵌入的跨类别距离，并利用知识蒸馏保持了 MHE 调整后和原始文本嵌入之间的相似性，以保留不同类别之间的语义信息。此外，该方法还引入了一种文本感知图像（TAI）调整机制，在训练阶段通过基于文本距离的自适应边界（TAM）增加了图像嵌入之间的跨类别距离，并利用知识蒸馏保持了 Fine-tuning 后和预训练图像嵌入之间的相似性。广泛的实验结果证明了该方法的有效性，在防御各种对抗扰动的同时，保持了原始 CLIP 模型的零 - shot 泛化能力。