This paper investigates the impact of different optimizers on the grokking phenomenon, where models exhibit delayed generalization. We conducted experiments across seven numerical tasks (primarily modular arithmetic) using a modern Transformer architecture. The experimental configuration systematically varied the optimizer (Muon vs. AdamW) and the softmax activation function (standard softmax, stablemax, and sparsemax) to assess their combined effect on learning dynamics. Our empirical evaluation reveals that the Muon optimizer, characterized by its use of spectral norm constraints and second-order information, significantly accelerates the onset of grokking compared to the widely used AdamW optimizer. Specifically, Muon reduced the mean grokking epoch from 153.09 to 102.89 across all configurations, a statistically significant difference (t = 5.0175, p = 6.33e-08). This suggests that the optimizer choice plays a crucial role in facilitating the transition from memorization to generalization.

本研究探讨了不同优化器对Grokking现象的影响，该现象指模型呈现出延迟泛化的特征。通过对七个数字任务的实验，发现缪子优化器通过引入谱范数约束和二阶信息，相比广泛使用的AdamW优化器显著加快了Grokking的发生，平均Grokking周期从153.09降低到102.89，表明优化器的选择在促进记忆与泛化之间的转变中起到了关键作用。

缪子优化器加速Grokking现象