Do neural networks, trained on well-understood algorithmic tasks, reliably
rediscover known algorithms for solving those tasks? Several recent studies, on
tasks ranging from group arithmetic to in-context linear regression, have
suggested that the answer is yes. Using modular addition as a prototypical
problem, we show that algorithm discovery in neural networks is sometimes more
complex. Small changes to model hyperparameters and initializations can induce
the discovery of qualitatively different algorithms from a fixed training set,
and even parallel implementations of multiple such algorithms. Some networks
trained to perform modular addition implement a familiar Clock algorithm;
others implement a previously undescribed, less intuitive, but comprehensible
procedure which we term the Pizza algorithm, or a variety of even more complex
procedures. Our results show that even simple learning problems can admit a
surprising diversity of solutions, motivating the development of new tools for
characterizing the behavior of neural networks across their algorithmic phase
space.

该研究研究发现，神经网络在一些算法任务上重新发现已知的算法是可靠的，但在模块加法等问题上，模型的超参数和初始化的微小变化可能会导致从固定数据集中发现不同的算法，有些神经网络实现的是熟悉的 Clock 算法，而另一些实现的是先前未描述的，不太直观但可以理解的 Pizza 算法，甚至包括更复杂的过程。