We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness. While deep networks are capable of memorizing noise data, our results suggest that they tend to prioritize learning simple patterns first. In our experiments, we expose qualitative differences in gradient-based optimization of deep neural networks (DNNs) on noise vs. real data. We also demonstrate that for appropriately tuned explicit regularization (e.g., dropout) we can degrade DNN training performance on noise datasets without compromising generalization on real data. Our analysis suggests that the notions of effective capacity which are dataset independent are unlikely to explain the generalization performance of deep networks when trained with gradient based methods because training data itself plays an important role in determining the degree of memorization.

本文研究深度学习中记忆化的作用，探讨了容量，泛化，对抗鲁棒性的联系。作者的实验表明，深度神经网络在优化噪声数据和真实数据时存在数量上的差异，但经过适当的正则化调整（如dropout），我们可以降低其在噪声数据集上的训练性能，同时不影响在真实数据上的泛化能力。研究结果表明，数据集本身在决定记忆化程度时具有重要作用，而基于梯度优化方法的深度网络的泛化性能不太可能由数据集独立的有效容量来解释。

深度网络内存化探究