Background. A main theoretical puzzle is why over-parameterized Neural
Networks (NNs) generalize well when trained to zero loss (i.e., so they
interpolate the data). Usually, the NN is trained with Stochastic Gradient
Descent (SGD) or one of its variants. However, recent empirical work examined
the generalization of a random NN that interpolates the data: the NN was
sampled from a seemingly uniform prior over the parameters, conditioned on that
the NN perfectly classifying the training set. Interestingly, such a NN sample
typically generalized as well as SGD-trained NNs.
Contributions. We prove that such a random NN interpolator typically
generalizes well if there exists an underlying narrow ``teacher NN" that agrees
with the labels. Specifically, we show that such a `flat' prior over the NN
parametrization induces a rich prior over the NN functions, due to the
redundancy in the NN structure. In particular, this creates a bias towards
simpler functions, which require less relevant parameters to represent --
enabling learning with a sample complexity approximately proportional to the
complexity of the teacher (roughly, the number of non-redundant parameters),
rather than the student's.

通过证明，我们发现基于狭窄的 “教师神经网络” 存在时，随机的神经网络插值器通常具有很好的泛化能力。具体而言，我们显示出这种对神经网络参数化的 “平坦” 先验在神经网络函数上引发丰富的先验，这是由于神经网络结构中的冗余性引起的。特别是，这会导致对于更简单的函数具有偏好，这些函数需要较少的相关参数来表示，以实现与教师相当数量的参数（大致上是非冗余参数的数量）相关的样本复杂度的学习。