Recently, an unusual phenomenon called grokking has gained much attention,
where sometimes a neural network generalizes long after it perfectly fits the
training data. We try to understand this seemingly strange phenomenon using the
robustness of the neural network. Using a robustness viewpoint, we show that
the popular $l_2$ weight norm (metric) of the neural network is actually a
sufficient condition for grokking. As we also empirically find that $l_2$ norm
correlates with grokking on the test data not in a timely way, we propose new
metrics based on robustness and information theory and find that our new
metrics correlate well with the grokking phenomenon. Based on the previous
observations, we propose methods to speed up the generalization process. In
addition, we examine the standard training process on modulo addition dataset
and find that it hardly learns other basic group operations before grokking,
including the commutative law. Interestingly, the speed up of generalization
when using our proposed method can be partially explained by learning the
commutative law, a necessary condition when the model groks on test dataset.

我们研究了神经网络中的一种异常现象叫做 “grokking”，发现使用 $l_2$ 权重范数和鲁棒性观点可以解释、测量和加速这一现象，同时研究发现在测试数据上，新的基于鲁棒性和信息理论的度量指标与 “grokking” 现象具有较好的相关性，并提出了提速泛化过程的方法。此外，我们还研究了基本群操作的学习过程，并发现在 “grokking” 之前，神经网络几乎没有学习到其他基本群操作，包括交换律。有趣的是，当使用我们提出的方法时，泛化过程加速的部分原因可以通过学习交换律来解释，这也是模型在测试数据集上实现 “grokking” 的必要条件。