Visual fine-tuning has garnered significant attention with the rise of
pre-trained vision models. The current prevailing method, full fine-tuning,
suffers from the issue of knowledge forgetting as it focuses solely on fitting
the downstream training set. In this paper, we propose a novel weight
rollback-based fine-tuning method called OLOR (One step Learning, One step
Review). OLOR combines fine-tuning with optimizers, incorporating a weight
rollback term into the weight update term at each step. This ensures
consistency in the weight range of upstream and downstream models, effectively
mitigating knowledge forgetting and enhancing fine-tuning performance. In
addition, a layer-wise penalty is presented to employ penalty decay and the
diversified decay rate to adjust the weight rollback levels of layers for
adapting varying downstream tasks. Through extensive experiments on various
tasks such as image classification, object detection, semantic segmentation,
and instance segmentation, we demonstrate the general applicability and
state-of-the-art performance of our proposed OLOR. Code is available at
this https URL

我们提出了一种名为 OLOR（One step Learning, One step Review）的新型基于权重回滚的微调方法，结合优化器，每一步都将权重回滚项引入权重更新项，确保上下游模型的权重范围一致，有效减轻知识遗忘问题，并提高微调性能。此外，我们还提出了一种逐层惩罚，利用衰减率调整层级的权重回滚水平，以适应不同的下游任务。通过对诸如图像分类、目标检测、语义分割和实例分割等各种任务的大量实验证明了我们提出的 OLOR 方法的广泛适用性和卓越性能。