Neural networks readily learn a subset of the modular arithmetic tasks, while
failing to generalize on the rest. This limitation remains unmoved by the
choice of architecture and training strategies. On the other hand, an
analytical solution for the weights of Multi-layer Perceptron (MLP) networks
that generalize on the modular addition task is known in the literature. In
this work, we (i) extend the class of analytical solutions to include modular
multiplication as well as modular addition with many terms. Additionally, we
show that real networks trained on these datasets learn similar solutions upon
generalization (grokking). (ii) We combine these "expert" solutions to
construct networks that generalize on arbitrary modular polynomials. (iii) We
hypothesize a classification of modular polynomials into learnable and
non-learnable via neural networks training; and provide experimental evidence
supporting our claims.

神经网络对模块化算术任务的学习受限，无法很好地进行推广；然而，在文献中已知有一种多层感知机（MLP）网络权重的解析解适用于模块化加法任务，本文将这种解析解的类别扩展到包括模块化乘法和具有多个项的模块化加法。此外，我们展示出在这些数据集上经过训练的真实网络通过泛化（理解）学习类似的解，我们结合这些 “专家” 解来构建在任意模块化多项式上具有推广性的网络，并猜测通过神经网络训练的模块化多项式可被分类为可学习和不可学习，并提供了支持我们观点的实验证据。

理解模块化多项式

Grokking Modular Polynomials

Large language models can solve tasks that were not present in the training
set. This capability is believed to be due to in-context learning and skill
composition. In this work, we study the emergence of in-context learning and
skill composition in a collection of modular arithmetic tasks. Specifically, we
consider a finite collection of linear modular functions $z = a \, x + b \, y
\;\mathrm{mod}\; p$ labeled by the vector $(a, b) \in \mathbb{Z}_p^2$. We use
some of these tasks for pre-training and the rest for out-of-distribution
testing. We empirically show that a GPT-style transformer exhibits a transition
from in-distribution to out-of-distribution generalization as the number of
pre-training tasks increases. We find that the smallest model capable of
out-of-distribution generalization requires two transformer blocks, while for
deeper models, the out-of-distribution generalization phase is
\emph{transient}, necessitating early stopping. Finally, we perform an
interpretability study of the pre-trained models, revealing the highly
structured representations in both phases; and discuss the learnt algorithm.

通过研究模块化算术任务中上下文学习和技能组合的出现，我们证明了 GPT-style transformer 模型从分布内到分布外泛化的转变，也发现了最小的能够进行分布外泛化的模型需要两个 transformer 块。