Influence functions aim to quantify the impact of individual training data points on a model's predictions. While extensive research has been conducted on influence functions in traditional machine learning models, their application to large language models (LLMs) has been limited. In this work, we conduct a systematic study to address a key question: do influence functions work on LLMs? Specifically, we evaluate influence functions across multiple tasks and find that they consistently perform poorly in most settings. Our further investigation reveals that their poor performance can be attributed to: (1) inevitable approximation errors when estimating the iHVP component due to the scale of LLMs, (2) uncertain convergence during fine-tuning, and, more fundamentally, (3) the definition itself, as changes in model parameters do not necessarily correlate with changes in LLM behavior. Our study thus suggests the need for alternative approaches for identifying influential samples. To support future work, our code is made available at https://github.com/plumprc/Failures-of-Influence-Functions-in-LLMs.

本研究针对影响函数在大型语言模型（LLM）中的应用进行了系统性研究，发现其在多数情况下表现不佳，原因包括近似误差、微调过程中的不确定收敛，以及模型参数变化与LLM行为之间缺乏关联。研究建议寻求替代方法以识别有影响力的样本，推动后续研究。

影响函数在大型语言模型上有效吗？