The success of modern neural networks has prompted study of the connection between memorisation and generalisation: overparameterised models generalise well, despite being able to perfectly fit (memorise) completely random labels. To carefully study this issue, Feldman proposed a metric to quantify the degree of memorisation of individual training examples, and empirically computed the corresponding memorisation profile of a ResNet on image classification bench-marks. While an exciting first glimpse into what real-world models memorise, this leaves open a fundamental question: do larger neural models memorise more? We present a comprehensive empirical analysis of this question on image classification benchmarks. We find that training examples exhibit an unexpectedly diverse set of memorisation trajectories across model sizes: most samples experience decreased memorisation under larger models, while the rest exhibit cap-shaped or increasing memorisation. We show that various proxies for the Feldman memorization score fail to capture these fundamental trends. Lastly, we find that knowledge distillation, an effective and popular model compression technique, tends to inhibit memorisation, while also improving generalisation. Specifically, memorisation is mostly inhibited on examples with increasing memorisation trajectories, thus pointing at how distillation improves generalisation.

现代神经网络的成功引发了对记忆和泛化之间关系的研究：过参数化的模型能够很好地进行泛化，尽管它们能够完美地拟合（记忆）完全随机的标签。本文通过对图像分类基准上ResNet的记忆度量进行实证计算，全面分析了神经模型大小对记忆能力的影响，结果发现不同模型大小的训练样本表现出意想不到的多样化记忆轨迹：较大模型下大多数样本减少了记忆，而部分样本呈现帽状或者递增型记忆。此外，我们发现知识蒸馏作为一种有效且流行的模型压缩技术，往往抑制记忆，同时提高了泛化能力，尤其是对于记录递增轨迹的示例，蒸馏技术能够有效提高泛化能力。

大型图像分类器记忆了什么？