Parameter-Efficient Fine-Tuning (PEFT) methods enable efficient adaptation of
Large Language Models (LLMs) to various downstream applications. However, the
effectiveness of the PEFT diminishes notably when downstream tasks require
accurate learning of factual knowledge. In this paper, we adopt a semantic
perspective to investigate this phenomenon, uncovering the reasons behind
PEFT's limitations in knowledge learning task. Our findings reveal that: (1)
PEFT presents a notable risk of pushing the model away from the intended
knowledge target; (2) multiple knowledge interfere with each other, and such
interference suppresses the learning and expression of knowledge features.
Based on these insights, we introduce a data filtering strategy to exclude data
that is detrimental to knowledge learning and a re-weighted learning strategy
to make the model attentive to semantic distance during knowledge learning.
Experimental results demonstrate the effectiveness of the proposed method on
open-source large language model, further validate the semantic challenge in
PEFT, thus paving the way for future research.

通过采用语义视角研究，揭示了 Parameter-Efficient Fine-Tuning（PEFT）方法在知识学习任务中的限制原因，并提出了数据过滤策略和语义距离加权学习策略，以提高模型对知识学习的关注度。实验证明了该方法在开源大语言模型上的有效性，并为未来的研究铺平道路。