Neural networks are overparametrized and easily overfit the datasets they
train on. In the extreme case, it is shown that they can memorize a training
set with fully randomized labels. We propose using the curvature of loss
function around the training sample as a measure of its memorization, averaged
over all training epochs. We use this to study the generalization versus
memorization properties of different samples in popular image datasets. We
visualize samples with the highest curvature of loss around them, and show that
these visually correspond to long-tailed, mislabeled or conflicting samples.
This analysis helps us find a, to the best of our knowledge, novel failure
model on the CIFAR100 dataset, that of duplicated images with different labels.
We also synthetically mislabel a proportion of the dataset by randomly
corrupting the labels of a few samples, and show that sorting by curvature
yields high AUROC values for identifying the mislabeled samples.

本文提出通过损失函数对训练样本曲率的测量来对神经网络的记忆化能力进行研究和分析。通过对 popular image datasets 的样本进行测量、可视化和排序，研究展示了神经网络在长尾数据、错误标签和冲突样本方面的记忆化失效，并发现了 CIFAR100 数据集中存在的重复标签问题。实验结果表明，通过曲率排序可以有效地找到疑似错误标签的样本。