With the prevalence of the Pretraining-Finetuning paradigm in transfer
learning, the robustness of downstream tasks has become a critical concern. In
this work, we delve into adversarial robustness in transfer learning and reveal
the critical role of initialization, including both the pretrained model and
the linear head. First, we discover the necessity of an adversarially robust
pretrained model. Specifically, we reveal that with a standard pretrained
model, Parameter-Efficient Finetuning~(PEFT) methods either fail to be
adversarially robust or continue to exhibit significantly degraded adversarial
robustness on downstream tasks, even with adversarial training during
finetuning. Leveraging a robust pretrained model, surprisingly, we observe that
a simple linear probing can outperform full finetuning and other PEFT methods
with random initialization on certain datasets. We further identify that linear
probing excels in preserving robustness from the robust pretraining. Based on
this, we propose Robust Linear Initialization~(RoLI) for adversarial
finetuning, which initializes the linear head with the weights obtained by
adversarial linear probing to maximally inherit the robustness from
pretraining. Across five different image classification datasets, we
demonstrate the effectiveness of RoLI and achieve new state-of-the-art results.

在这项研究中，我们深入研究了迁移学习中的对抗鲁棒性，并揭示了初始化的关键作用，包括预训练模型和线性头部。我们发现对抗鲁棒的预训练模型的必要性，并通过线性探测发现它能在某些数据集上优于全面微调和其他方法。基于此，我们提出了 Robust Linear Initialization（RoLI）来对抗微调，通过使用对抗线性探测获得的权重来最大程度地继承预训练的鲁棒性，并在五个不同的图像分类数据集上取得了新的最优结果。