Quantifying the impact of training data points is crucial for understanding
the outputs of machine learning models and for improving the transparency of
the AI pipeline. The influence function is a principled and popular data
attribution method, but its computational cost often makes it challenging to
use. This issue becomes more pronounced in the setting of large language models
and text-to-image models. In this work, we propose DataInf, an efficient
influence approximation method that is practical for large-scale generative AI
models. Leveraging an easy-to-compute closed-form expression, DataInf
outperforms existing influence computation algorithms in terms of computational
and memory efficiency. Our theoretical analysis shows that DataInf is
particularly well-suited for parameter-efficient fine-tuning techniques such as
LoRA. Through systematic empirical evaluations, we show that DataInf accurately
approximates influence scores and is orders of magnitude faster than existing
methods. In applications to RoBERTa-large, Llama-2-13B-chat, and
stable-diffusion-v1.5 models, DataInf effectively identifies the most
influential fine-tuning examples better than other approximate influence
scores. Moreover, it can help to identify which data points are mislabeled.

我们提出了 DataInf，一种高效的影响力近似计算方法，可用于大规模生成型 AI 模型。通过利用易于计算的闭合形式表达式，DataInf 在计算和内存效率方面优于现有的影响力计算算法。我们的理论分析表明，DataInf 特别适用于 LoRA 等参数效率高的微调技术。通过系统的实证评估，我们表明 DataInf 准确地近似了影响力分数，并且比现有方法快几个数量级。在 RoBERTa-large、Llama-2-13B-chat 和 stable-diffusion-v1.5 模型的应用中，DataInf 能够比其他近似影响力分数更好地识别出影响最大的微调示例，而且可以帮助识别出错误标记的数据点。