Catastrophic Forgetting (CF) means models forgetting previously acquired
knowledge when learning new data. It compromises the effectiveness of large
language models (LLMs) during fine-tuning, yet the underlying causes have not
been thoroughly investigated. This paper takes the first step to reveal the
direct link between the flatness of the model loss landscape and the extent of
CF in the field of LLMs. Based on this, we introduce the sharpness-aware
minimization to mitigate CF by flattening the loss landscape. Experiments on
three widely-used fine-tuning datasets, spanning different model scales,
demonstrate the effectiveness of our method in alleviating CF. Analyses show
that we nicely complement the existing anti-forgetting strategies, further
enhancing the resistance of LLMs to CF.

本文首次揭示了大型语言模型（LLMs）的模型损失面与遗忘程度之间的直接联系，并引入了锐度感知最小化方法来减轻 Catastrophic Forgetting，通过使损失面变得平坦。实验证明该方法在缓解 Catastrophic Forgetting 方面的效果显著，并且通过与现有的抗遗忘策略的结合，进一步增强了 LLMs 的抵抗能力。