Pre-trained large language models (LLMs) are commonly fine-tuned to adapt to downstream tasks. Since the majority of knowledge is acquired during pre-training, attributing the predictions of fine-tuned LLMs to their pre-training data may provide valuable insights. Influence functions have been proposed as a means to explain model predictions based on training data. However, existing approaches fail to compute ``multi-stage'' influence and lack scalability to billion-scale LLMs. In this paper, we propose the multi-stage influence function to attribute the downstream predictions of fine-tuned LLMs to pre-training data under the full-parameter fine-tuning paradigm. To enhance the efficiency and practicality of our multi-stage influence function, we leverage Eigenvalue-corrected Kronecker-Factored (EK-FAC) parameterization for efficient approximation. Empirical results validate the superior scalability of EK-FAC approximation and the effectiveness of our multi-stage influence function. Additionally, case studies on a real-world LLM, dolly-v2-3b, demonstrate its interpretive power, with exemplars illustrating insights provided by multi-stage influence estimates. Our code is public at https://github.com/colored-dye/multi_stage_influence_function.

本研究解决了现有多阶段影响函数在对大规模预训练大型语言模型（LLM）进行解释时缺乏可扩展性的问题。我们提出的多阶段影响函数，结合特征值校正的Kronecker因子化参数化，能够有效地归因于下游任务的预测并提高计算效率。实证结果表明，该方法在处理大规模模型时具有优越的可扩展性和解释能力，对理解模型预测提供了重要的见解。

通过特征值校正的Kronecker因子化参数化实现可扩展的多阶段影响函数