Fine-tuning is becoming widely used for leveraging the power of pre-trained
foundation models in new downstream tasks. While there are many successes of
fine-tuning on various tasks, recent studies have observed challenges in the
generalization of fine-tuned models to unseen distributions (i.e.,
out-of-distribution; OOD). To improve OOD generalization, some previous studies
identify the limitations of fine-tuning data and regulate fine-tuning to
preserve the general representation learned from pre-training data. However,
potential limitations in the pre-training data and models are often ignored. In
this paper, we contend that overly relying on the pre-trained representation
may hinder fine-tuning from learning essential representations for downstream
tasks and thus hurt its OOD generalization. It can be especially catastrophic
when new tasks are from different (sub)domains compared to pre-training data.
To address the issues in both pre-training and fine-tuning data, we propose a
novel generalizable fine-tuning method LEVI, where the pre-trained model is
adaptively ensembled layer-wise with a small task-specific model, while
preserving training and inference efficiencies. By combining two complementing
models, LEVI effectively suppresses problematic features in both the
fine-tuning data and pre-trained model and preserves useful features for new
tasks. Broad experiments with large language and vision models show that LEVI
greatly improves fine-tuning generalization via emphasizing different views
from fine-tuning data and pre-trained features.

基于已训练好的基础模型进行微调在新的下游任务中得到广泛应用，但是存在着泛化到未见分布（即超过分布；OOD）的挑战。为了改善 OOD 的泛化能力，本文提出了一种新的通用微调方法 LEVI，在保留训练和推断效率的同时，通过自适应地对预训练模型进行逐层集成和与小型任务专属模型相结合，有效地抑制了微调数据和预训练模型中的问题特征，保留了新任务所需的有用特征。通过大规模语言和视觉模型进行的广泛实验表明，LEVI 通过强调来自微调数据和预训练特征的不同视角，极大地提高了微调的泛化能力。