Pre-trained foundation models, owing primarily to their enormous capacity and
exposure to vast amount of training data scraped from the internet, enjoy the
advantage of storing knowledge about plenty of real-world concepts. Such models
are typically fine-tuned on downstream datasets to produce remarkable
state-of-the-art performances. While various fine-tuning methods have been
devised and are shown to be highly effective, we observe that a fine-tuned
model's ability to recognize concepts on tasks $\textit{different}$ from the
downstream one is reduced significantly compared to its pre-trained
counterpart. This is clearly undesirable as a huge amount of time and money
went into learning those very concepts in the first place. We call this
undesirable phenomenon "concept forgetting" and via experiments show that most
end-to-end fine-tuning approaches suffer heavily from this side effect. To this
end, we also propose a rather simple fix to this problem by designing a method
called LDIFS (short for $\ell_2$ distance in feature space) that simply
preserves the features of the original foundation model during fine-tuning. We
show that LDIFS significantly reduces concept forgetting without having
noticeable impact on the downstream task performance.

通过使用 LDIFS 方法，可以在保持下游任务性能不受明显影响的情况下，显著减少概念遗忘现象。