Vision-language models (VLMs) have achieved significant strides in recent times specially in multimodal tasks, yet they remain susceptible to adversarial attacks on their vision components. To address this, we propose Sim-CLIP, an unsupervised adversarial fine-tuning method that enhances the robustness of the widely-used CLIP vision encoder against such attacks while maintaining semantic richness and specificity. By employing a Siamese architecture with cosine similarity loss, Sim-CLIP learns semantically meaningful and attack-resilient visual representations without requiring large batch sizes or momentum encoders. Our results demonstrate that VLMs enhanced with Sim-CLIP's fine-tuned CLIP encoder exhibit significantly enhanced robustness against adversarial attacks, while preserving semantic meaning of the perturbed images. Notably, Sim-CLIP does not require additional training or fine-tuning of the VLM itself; replacing the original vision encoder with our fine-tuned Sim-CLIP suffices to provide robustness. This work underscores the significance of reinforcing foundational models like CLIP to safeguard the reliability of downstream VLM applications, paving the way for more secure and effective multimodal systems.

本研究针对视觉语言模型在多模态任务中易受对抗攻击这一问题，提出了Sim-CLIP，这是一个无监督的对抗微调方法，旨在增强CLIP视觉编码器的鲁棒性，同时保持语义丰富性。研究表明，采用Sim-CLIP微调的CLIP编码器能显著提高视觉语言模型对对抗攻击的抗性，并且无需额外的训练步骤，简单替换编码器即可提高鲁棒性。

Sim-CLIP：无监督的Siamese对抗微调用于增强视觉语言模型的鲁棒性和语义丰富性