Vision-Language Models (VLMs) are a significant technique for Artificial
General Intelligence (AGI). With the fast growth of AGI, the security problem
become one of the most important challenges for VLMs. In this paper, through
extensive experiments, we demonstrate the vulnerability of the conventional
adaptation methods for VLMs, which may bring significant security risks. In
addition, as the size of the VLMs increases, performing conventional
adversarial adaptation techniques on VLMs results in high computational costs.
To solve these problems, we propose a parameter-efficient
\underline{Adv}ersarial adaptation method named \underline{AdvLoRA} by
\underline{Lo}w-\underline{R}ank \underline{A}daptation. At first, we
investigate and reveal the intrinsic low-rank property during the adversarial
adaptation for VLMs. Different from LoRA, we improve the efficiency and
robustness of adversarial adaptation by designing a novel reparameterizing
method based on parameter clustering and parameter alignment. In addition, an
adaptive parameter update strategy is proposed to further improve the
robustness. By these settings, our proposed AdvLoRA alleviates the model
security and high resource waste problems. Extensive experiments demonstrate
the effectiveness and efficiency of the AdvLoRA.

通过大量实验，我们展示了传统的视觉 - 语言模型（VLMs）所采用的常规适应方法的脆弱性，这可能带来重大的安全风险。此外，随着 VLMs 规模的增加，对 VLMs 进行传统的对抗性适应技术会带来高计算成本。为了解决这些问题，我们提出了一种名为 AdvLoRA 的参数高效对抗性适应方法，通过低秩适应进行了改进，设计了一种基于参数聚类和参数对齐的新型重新参数化方法，进一步提高了鲁棒性，解决了模型安全性和资源浪费问题。通过大量实验证明了 AdvLoRA 的有效性和高效性。