When evaluating the performance of a pre-trained model transferred to a downstream task, it is imperative to assess not only the in-distribution (ID) accuracy of the downstream model but also its capacity to generalize and identify out-of-distribution (OOD) samples. In this paper, we unveil the hidden costs associated with intrusive fine-tuning techniques. Specifically, we demonstrate that commonly used fine-tuning methods not only distort the representations necessary for generalizing to covariate-shifted OOD samples (OOD generalization) but also distort the representations necessary for detecting semantically-shifted OOD samples (OOD detection). To address these challenges, we introduce a new model reprogramming approach for fine-tuning, which we name Reprogrammer. Reprogrammer aims to improve the holistic performance of the downstream model across ID, OOD generalization, and OOD detection tasks. Our empirical evidence reveals that Reprogrammer is less intrusive and yields superior downstream models. Furthermore, we demonstrate that by appending an additional representation residual connection to Reprogrammer, we can further preserve pre-training representations, resulting in an even more safe and robust downstream model capable of excelling in many ID classification, OOD generalization, and OOD detection settings.

评估预训练模型在下游任务中的性能时，除了评估下游模型的内分布准确性外，还必须评估其对于一般化和识别离群样本的能力。本文揭示了侵入性微调技术所带来的隐藏成本，提出了Reprogrammer方法来改善下游模型在内分布、离群泛化和离群检测任务中的整体性能。实证证据表明Reprogrammer是非侵入性的，能产生更优秀的下游模型，并通过添加额外的表示残差连接进一步保留预训练表示，从而获得更安全、更强大的下游模型，能在许多内分布分类、离群泛化和离群检测设置中表现出色。

模型重新编程在文本-图像编码器的非分布数据上优于微调