It is well known that neural networks with many more parameters than training
examples do not overfit. Implicit regularization phenomena, which are still not
well understood, occur during optimization and 'good' networks are favored.
Thus the number of parameters is not an adequate measure of complexity if we do
not consider all possible networks but only the 'good' ones. To better
understand which networks are favored during optimization, we study the
geometry of the output set as parameters vary. When the inputs are fixed, we
prove that the dimension of this set changes and that the local dimension,
called batch functional dimension, is almost surely determined by the
activation patterns in the hidden layers. We prove that the batch functional
dimension is invariant to the symmetries of the network parameterization:
neuron permutations and positive rescalings. Empirically, we establish that the
batch functional dimension decreases during optimization. As a consequence,
optimization leads to parameters with low batch functional dimensions. We call
this phenomenon geometry-induced implicit regularization.The batch functional
dimension depends on both the network parameters and inputs. To understand the
impact of the inputs, we study, for fixed parameters, the largest attainable
batch functional dimension when the inputs vary. We prove that this quantity,
called computable full functional dimension, is also invariant to the
symmetries of the network's parameterization, and is determined by the
achievable activation patterns. We also provide a sampling theorem, showing a
fast convergence of the estimation of the computable full functional dimension
for a random input of increasing size. Empirically we find that the computable
full functional dimension remains close to the number of parameters, which is
related to the notion of local identifiability. This differs from the observed
values for the batch functional dimension computed on training inputs and test
inputs. The latter are influenced by geometry-induced implicit regularization.

神经网络的隐式正则化现象与参数的几何特征密切相关，优化过程中更倾向于具有低批次功能维度的参数。网络的输入也对批次功能维度产生影响，输入的变化会改变批次功能维度的上限，称为可计算全功能维度，其估计收敛迅速且保持接近参数个数，可以与局部可辨识性相对应。与训练输入和测试输入相关的批次功能维度则由几何诱导的隐式正则化所影响。