Understanding the internal representations learned by neural networks is a cornerstone challenge in the science of machine learning. While there have been significant recent strides in some cases towards understanding how neural networks implement specific target functions, this paper explores a complementary question -- why do networks arrive at particular computational strategies? Our inquiry focuses on the algebraic learning tasks of modular addition, sparse parities, and finite group operations. Our primary theoretical findings analytically characterize the features learned by stylized neural networks for these algebraic tasks. Notably, our main technique demonstrates how the principle of margin maximization alone can be used to fully specify the features learned by the network. Specifically, we prove that the trained networks utilize Fourier features to perform modular addition and employ features corresponding to irreducible group-theoretic representations to perform compositions in general groups, aligning closely with the empirical observations of Nanda et al. and Chughtai et al. More generally, we hope our techniques can help to foster a deeper understanding of why neural networks adopt specific computational strategies.

理解神经网络学习到的内部表示是机器学习科学中的一个基石挑战。本文探讨了一个互补问题，即为什么网络会采用特定的计算策略，通过代数学习任务对用于这些任务的网络学到的特征进行了理论分析，证明了训练网络利用傅立叶特征进行模块化加法，以及使用与不可约群表示对应的特征来执行一般群的组合，与Nanda等人和Chughtai等人的实证观察密切一致。

通过边缘最大化实现特征的出现：代数任务中的案例研究