What has an Artificial Neural Network (ANN) learned after being successfully
trained to solve a task - the set of training items or the relations between
them? This question is difficult to answer for modern applied ANNs because of
their enormous size and complexity. Therefore, here we consider a
low-dimensional network and a simple task, i.e., the network has to reproduce a
set of training items identically. We construct the family of solutions
analytically and use standard learning algorithms to obtain numerical
solutions. These numerical solutions differ depending on the optimization
algorithm and the weight initialization and are shown to be particular members
of the family of analytical solutions. In this simple setting, we observe that
the general structure of the network weights represents the training set's
symmetry group, i.e., the relations between training items. As a consequence,
linear networks generalize, i.e., reproduce items that were not part of the
training set but are consistent with the symmetry of the training set. In
contrast, non-linear networks tend to learn individual training items and show
associative memory. At the same time, their ability to generalize is limited. A
higher degree of generalization is obtained for networks whose activation
function contains a linear regime, such as tanh. Our results suggest ANN's
ability to generalize - instead of learning items - could be improved by
generating a sufficiently big set of elementary operations to represent
relations and strongly depends on the applied non-linearity.

人工神经网络（ANN）通过成功训练来解决任务后，学到的是训练项集还是它们之间的关系？在现代应用的 ANN 中，这个问题很难回答，因为其规模和复杂性巨大。因此，我们在这里考虑一个低维网络和一个简单任务，即网络必须完全复制一系列训练项。我们通过分析构建了解析解的解集家族，并使用标准的学习算法得到数值解。这些数值解根据优化算法和权重初始化的不同而不同，并且被证明是解析解解集的特定成员。在这个简单的设置中，我们观察到网络权重的一般结构表示训练集的对称群，即训练项之间的关系。因此，线性网络具有泛化能力，即可以复制不属于训练集但与训练集的对称性一致的项。相反，非线性网络倾向于学习各个训练项，并显示出联想记忆的特点。同时，它们的泛化能力有限。具有包含线性区域的激活函数（如 tanh）的网络具有更高程度的泛化能力。我们的结果表明，ANN 的泛化能力可以通过生成足够大的基本操作集来表示关系，并且它强烈依赖于应用的非线性特性而得到改善。