One puzzling artifact in machine learning dubbed grokking is where delayed
generalization is achieved tenfolds of iterations after near perfect
overfitting to the training data. Focusing on the long delay itself on behalf
of machine learning practitioners, our goal is to accelerate generalization of
a model under grokking phenomenon. By regarding a series of gradients of a
parameter over training iterations as a random signal over time, we can
spectrally decompose the parameter trajectories under gradient descent into two
components: the fast-varying, overfitting-yielding component and the
slow-varying, generalization-inducing component. This analysis allows us to
accelerate the grokking phenomenon more than $\times 50$ with only a few lines
of code that amplifies the slow-varying components of gradients. The
experiments show that our algorithm applies to diverse tasks involving images,
languages, and graphs, enabling practical availability of this peculiar
artifact of sudden generalization. Our code is available at
https://github.com/ironjr/grokfast.

对于机器学习中的一个令人困惑的伪现象，即迟滞推广现象，我们通过将参数的一系列梯度作为随时间变化的时间随机信号进行频谱分解，加速了该现象，并提出了少量代码来放大梯度的缓慢变化部分，其中实验表明我们的算法适用于涉及图像、语言和图形的各种任务，使得这种突然泛化现象更具实际可行性。

Grokfast：通过增强慢梯度加速洞察力的理解

Grokfast: Accelerated Grokking by Amplifying Slow Gradients

Grokking has been actively explored to reveal the mystery of delayed
generalization. Identifying interpretable algorithms inside the grokked models
is a suggestive hint to understanding its mechanism. In this work, beyond the
simplest and well-studied modular addition, we observe the internal circuits
learned through grokking in complex modular arithmetic via interpretable
reverse engineering, which highlights the significant difference in their
dynamics: subtraction poses a strong asymmetry on Transformer; multiplication
requires cosine-biased components at all the frequencies in a Fourier domain;
polynomials often result in the superposition of the patterns from elementary
arithmetic, but clear patterns do not emerge in challenging cases; grokking can
easily occur even in higher-degree formulas with basic symmetric and
alternating expressions. We also introduce the novel progress measure for
modular arithmetic; Fourier Frequency Sparsity and Fourier Coefficient Ratio,
which not only indicate the late generalization but also characterize
distinctive internal representations of grokked models per modular operation.
Our empirical analysis emphasizes the importance of holistic evaluation among
various combinations.

通过解释性的逆向工程，我们观察到通过 Grokking 在复杂模块化算术中学习到的内部电路，强调了它们动力学的显着差异，此外我们引入了模块化算术的新进展衡量以及可识别模型的内部表示。

复杂模块算术中解释曲解的变形金刚

Interpreting Grokked Transformers in Complex Modular Arithmetic

Grokking, or delayed generalization, is a phenomenon where generalization in
a deep neural network (DNN) occurs long after achieving near zero training
error. Previous studies have reported the occurrence of grokking in specific
controlled settings, such as DNNs initialized with large-norm parameters or
transformers trained on algorithmic datasets. We demonstrate that grokking is
actually much more widespread and materializes in a wide range of practical
settings, such as training of a convolutional neural network (CNN) on CIFAR10
or a Resnet on Imagenette. We introduce the new concept of delayed robustness,
whereby a DNN groks adversarial examples and becomes robust, long after
interpolation and/or generalization. We develop an analytical explanation for
the emergence of both delayed generalization and delayed robustness based on a
new measure of the local complexity of a DNN's input-output mapping. Our local
complexity measures the density of the so-called 'linear regions' (aka, spline
partition regions) that tile the DNN input space, and serves as a utile
progress measure for training. We provide the first evidence that for
classification problems, the linear regions undergo a phase transition during
training whereafter they migrate away from the training samples (making the DNN
mapping smoother there) and towards the decision boundary (making the DNN
mapping less smooth there). Grokking occurs post phase transition as a robust
partition of the input space emerges thanks to the linearization of the DNN
mapping around the training points. Website: this https URL

深度神经网络的一项研究探索了 “Grokking” 或延迟泛化的现象及其对网络鲁棒性和局部复杂性的影响。