In recent years, the Vision Transformer (ViT) model has gradually become
mainstream in various computer vision tasks, and the robustness of the model
has received increasing attention. However, existing large models tend to
prioritize performance during training, potentially neglecting the robustness,
which may lead to serious security concerns. In this paper, we establish a new
challenge: exploring how to use a small number of additional parameters for
adversarial finetuning to quickly and effectively enhance the adversarial
robustness of a standardly trained model. To address this challenge, we develop
the novel LNLoRA module, incorporating a learnable layer normalization before
the conventional LoRA module, which helps mitigate magnitude differences in
parameters between the adversarial and standard training paradigms.
Furthermore, we propose the FullLoRA-AT framework by integrating the
learnable LNLoRA modules into all key components of ViT-based models while
keeping the pretrained model frozen, which can significantly improve the model
robustness via adversarial finetuning in a parameter-efficient manner.
Extensive experiments on CIFAR-10, CIFAR-100, and Imagenette demonstrate the
superiority of our proposed FullLoRA-AT framework. It achieves comparable
robustness with full finetuning while only requiring about 5% of the learnable
parameters. This also effectively addresses concerns regarding extra model
storage space and enormous training time caused by adversarial finetuning.

通过使用学习性的 LNLoRA 模块和 FullLoRA-AT 框架，本文提出了一种参数高效的对 Vision Transformer 模型进行对抗微调，以有效提高模型的鲁棒性。在 CIFAR-10，CIFAR-100 和 Imagenette 数据集上进行的广泛实验证明了本文方法的优越性。