Handling distribution shifts from training data, known as out-of-distribution (OOD) generalization, poses a significant challenge in the field of machine learning. While a pre-trained vision-language model like CLIP has demonstrated remarkable zero-shot performance, further adaptation of the model to downstream tasks leads to undesirable degradation for OOD data. In this work, we introduce Sparse Adaptation for Fine-Tuning (SAFT), a method that prevents fine-tuning from forgetting the general knowledge in the pre-trained model. SAFT only updates a small subset of important parameters whose gradient magnitude is large, while keeping the other parameters frozen. SAFT is straightforward to implement and conceptually simple. Extensive experiments show that with only 0.1% of the model parameters, SAFT can significantly improve the performance of CLIP. It consistently outperforms baseline methods across several benchmarks. On the few-shot learning benchmark of ImageNet and its variants, SAFT gives a gain of 5.15% on average over the conventional fine-tuning method in OOD settings.

本文介绍了一种名为Sparse Adaptation for Fine-Tuning（SAFT）的方法，它通过只更新一小部分重要参数来防止微调过程中遗忘预训练模型的通用知识。实验证明，使用SAFT可以显著提升CLIP模型的性能，在多个基准任务中，SAFT方法始终优于基准方法，在OOD环境中，与传统微调方法相比，SAFT平均提升5.15%。

SAFT: 朝向微调的超出分布泛化