The parameter space for any fixed architecture of feedforward ReLU neural
networks serves as a proxy during training for the associated class of
functions - but how faithful is this representation? It is known that many
different parameter settings can determine the same function. Moreover, the
degree of this redundancy is inhomogeneous: for some networks, the only
symmetries are permutation of neurons in a layer and positive scaling of
parameters at a neuron, while other networks admit additional hidden
symmetries. In this work, we prove that, for any network architecture where no
layer is narrower than the input, there exist parameter settings with no hidden
symmetries. We also describe a number of mechanisms through which hidden
symmetries can arise, and empirically approximate the functional dimension of
different network architectures at initialization. These experiments indicate
that the probability that a network has no hidden symmetries decreases towards
0 as depth increases, while increasing towards 1 as width and input dimension
increase.

研究了前馈 ReLU 神经网络架构的参数空间，证明了对于没有窄于输入层的任何网络架构，都存在没有隐藏对称性的参数设置，并通过实验近似计算了不同网络架构在初始化时的功能维度。