This paper explores the implicit bias of overparameterized neural networks of
depth greater than two layers. Our framework considers a family of networks of
varying depths that all have the same capacity but different implicitly defined
representation costs. The representation cost of a function induced by a neural
network architecture is the minimum sum of squared weights needed for the
network to represent the function; it reflects the function space bias
associated with the architecture. Our results show that adding linear layers to
a ReLU network yields a representation cost that favors functions that can be
approximated by a low-rank linear operator composed with a function with low
representation cost using a two-layer network. Specifically, using a neural
network to fit training data with minimum representation cost yields an
interpolating function that is nearly constant in directions orthogonal to a
low-dimensional subspace. This means that the learned network will
approximately be a single- or multiple-index model. Our experiments show that
when this active subspace structure exists in the data, adding linear layers
can improve generalization and result in a network that is well-aligned with
the true active subspace.

本文探讨了超参数神经网络中大于两层的隐式偏差。通过添加线性层，可以优化神经网络的表示成本，并提高实际子空间的准确匹配度与预测性能。