When evaluating the performance of a pre-trained model transferred to a
downstream task, it is imperative to assess not only the in-distribution (ID)
accuracy of the downstream model but also its capacity to generalize and
identify out-of-distribution (OOD) samples. In this paper, we unveil the hidden
costs associated with intrusive fine-tuning techniques. Specifically, we
demonstrate that commonly used fine-tuning methods not only distort the
representations necessary for generalizing to covariate-shifted OOD samples
(OOD generalization) but also distort the representations necessary for
detecting semantically-shifted OOD samples (OOD detection). To address these
challenges, we introduce a new model reprogramming approach for fine-tuning,
which we name Reprogrammer. Reprogrammer aims to improve the holistic
performance of the downstream model across ID, OOD generalization, and OOD
detection tasks. Our empirical evidence reveals that Reprogrammer is less
intrusive and yields superior downstream models. Furthermore, we demonstrate
that by appending an additional representation residual connection to
Reprogrammer, we can further preserve pre-training representations, resulting
in an even more safe and robust downstream model capable of excelling in many
ID classification, OOD generalization, and OOD detection settings.

评估预训练模型在下游任务中的性能时，除了评估下游模型的内分布准确性外，还必须评估其对于一般化和识别离群样本的能力。本文揭示了侵入性微调技术所带来的隐藏成本，提出了 Reprogrammer 方法来改善下游模型在内分布、离群泛化和离群检测任务中的整体性能。实证证据表明 Reprogrammer 是非侵入性的，能产生更优秀的下游模型，并通过添加额外的表示残差连接进一步保留预训练表示，从而获得更安全、更强大的下游模型，能在许多内分布分类、离群泛化和离群检测设置中表现出色。