Robust Fine-Tuning (RFT) is a low-cost strategy to obtain adversarial
robustness in downstream applications, without requiring a lot of computational
resources and collecting significant amounts of data. This paper uncovers an
issue with the existing RFT, where optimizing both adversarial and natural
objectives through the feature extractor (FE) yields significantly divergent
gradient directions. This divergence introduces instability in the optimization
process, thereby hindering the attainment of adversarial robustness and
rendering RFT highly sensitive to hyperparameters. To mitigate this issue, we
propose a low-rank (LoRa) branch that disentangles RFT into two distinct
components: optimizing natural objectives via the LoRa branch and adversarial
objectives via the FE. Besides, we introduce heuristic strategies for
automating the scheduling of the learning rate and the scalars of loss terms.
Extensive empirical evaluations demonstrate that our proposed automated RFT
disentangled via the LoRa branch (AutoLoRa) achieves new state-of-the-art
results across a range of downstream tasks. AutoLoRa holds significant
practical utility, as it automatically converts a pre-trained FE into an
adversarially robust model for downstream tasks without the need for searching
hyperparameters.

在该论文中，我们揭示了现有 Robust Fine-Tuning 方法存在的问题，即通过特征提取器同时优化对抗目标和自然目标会导致梯度方向明显偏离，从而降低对抗鲁棒性，并使得 Robust Fine-Tuning 非常敏感于超参数。为了解决这个问题，我们提出了通过低秩支路将 Robust Fine-Tuning 分解为两个独立的部分来优化自然目标和对抗目标。此外，我们引入了启发式策略来自动调整学习率和损失项的标量。大量的实证评估表明，我们提出的通过低秩支路自动分解的 Robust Fine-Tuning（AutoLoRa）在各种下游任务中取得了最新的最优结果。AutoLoRa 具有重要的实用性，因为它可以将预训练的特征提取器自动转换为对抗鲁棒的下游任务模型，无需搜索超参数。