Contrastive vision-language models, e.g., CLIP, have garnered substantial
attention for their exceptional generalization capabilities. However, their
robustness to perturbations has ignited concerns. Existing strategies typically
reinforce their resilience against adversarial examples by enabling the image
encoder to "see" these perturbed examples, often necessitating a complete
retraining of the image encoder on both natural and adversarial samples. In
this study, we propose a new method to enhance robustness solely through text
augmentation, eliminating the need for retraining the image encoder on
adversarial examples. Our motivation arises from the realization that text and
image data inherently occupy a shared latent space, comprising latent content
variables and style variables. This insight suggests the feasibility of
learning to disentangle these latent content variables using text data
exclusively. To accomplish this, we introduce an effective text augmentation
method that focuses on modifying the style while preserving the content in the
text data. By changing the style part of the text data, we empower the text
encoder to emphasize latent content variables, ultimately enhancing the
robustness of vision-language models. Our experiments across various datasets
demonstrate substantial improvements in the robustness of the pre-trained CLIP
model.

通过文本增强方法，不需要在对抗性示例上重新训练图像编码器，从而增强视觉 - 语言模型的稳健性，并且实验证明了在各种数据集上对预训练的 CLIP 模型的稳健性有显著改善。