Influence functions serve as crucial tools for assessing sample influence in
model interpretation, subset training set selection, noisy label detection, and
more. By employing the first-order Taylor extension, influence functions can
estimate sample influence without the need for expensive model retraining.
However, applying influence functions directly to deep models presents
challenges, primarily due to the non-convex nature of the loss function and the
large size of model parameters. This difficulty not only makes computing the
inverse of the Hessian matrix costly but also renders it non-existent in some
cases. Various approaches, including matrix decomposition, have been explored
to expedite and approximate the inversion of the Hessian matrix, with the aim
of making influence functions applicable to deep models. In this paper, we
revisit a specific, albeit naive, yet effective approximation method known as
TracIn. This method substitutes the inverse of the Hessian matrix with an
identity matrix. We provide deeper insights into why this simple approximation
method performs well. Furthermore, we extend its applications beyond measuring
model utility to include considerations of fairness and robustness. Finally, we
enhance TracIn through an ensemble strategy. To validate its effectiveness, we
conduct experiments on synthetic data and extensive evaluations on noisy label
detection, sample selection for large language model fine-tuning, and defense
against adversarial attacks.

借助第一阶泰勒展开，影响函数可以估计样本对模型的影响力，无需进行昂贵的模型重新训练；本文通过探索矩阵分解等方法加快和近似海森矩阵的求逆过程，将影响函数应用于深度模型，并提出了一种称为 TracIn 的简单逼近方法，该方法通过将海森矩阵的逆替换为单位矩阵，不仅性能良好，还将应用拓展到模型公平性和鲁棒性领域，并通过集成策略改进了 TracIn 的效果；通过在合成数据上进行实验以及深入评估噪声标签检测、大型语言模型微调的样本选择和对抗攻击防御等方面验证了其有效性。

重审，扩展和增强无 Hessian 函数的影响

Revisit, Extend, and Enhance Hessian-Free Influence Functions

This paper presents a new way to identify additional positive pairs for BYOL,
a state-of-the-art (SOTA) self-supervised learning framework, to improve its
representation learning ability. Unlike conventional BYOL which relies on only
one positive pair generated by two augmented views of the same image, we argue
that information from different images with the same label can bring more
diversity and variations to the target features, thus benefiting representation
learning. To identify such pairs without any label, we investigate TracIn, an
instance-based and computationally efficient influence function, for BYOL
training. Specifically, TracIn is a gradient-based method that reveals the
impact of a training sample on a test sample in supervised learning. We extend
it to the self-supervised learning setting and propose an efficient batch-wise
per-sample gradient computation method to estimate the pairwise TracIn to
represent the similarity of samples in the mini-batch during training. For each
image, we select the most similar sample from other images as the additional
positive and pull their features together with BYOL loss. Experimental results
on two public medical datasets (i.e., ISIC 2019 and ChestX-ray) demonstrate
that the proposed method can improve the classification performance compared to
other competitive baselines in both semi-supervised and transfer learning
settings.

本文提出了一种新的方法，即使用 TracIn 作为实例基础的、计算有效的影响函数来为 BYOL 自我监督学习框架识别额外的正对，并通过 TracIn 估计成对 TracIn 来表示小批量中样本的相似度，以改善其表征学习能力，实验结果表明该方法可以在半监督和迁移学习设置下改善分类性能。

额外正向样本促进医学图像的更好表示学习

Additional Positive Enables Better Representation Learning for Medical  Images

We introduce a method called TracIn that computes the influence of a training
example on a prediction made by the model. The idea is to trace how the loss on
the test point changes during the training process whenever the training
example of interest was utilized. We provide a scalable implementation of
TracIn via: (a) a first-order gradient approximation to the exact computation,
(b) saved checkpoints of standard training procedures, and (c) cherry-picking
layers of a deep neural network. In contrast with previously proposed methods,
TracIn is simple to implement; all it needs is the ability to work with
gradients, checkpoints, and loss functions. The method is general. It applies
to any machine learning model trained using stochastic gradient descent or a
variant of it, agnostic of architecture, domain and task. We expect the method
to be widely useful within processes that study and improve training data.

我们介绍了一种名为 TracIn 的方法，该方法通过跟踪训练示例对模型预测的影响来计算模型的影响。与以前提出的方法相比，TracIn 简单易实现，适用于任何使用随机梯度下降或其变体训练的机器学习模型，且可广泛应用于研究和改进训练数据的过程。