It is well known that (stochastic) gradient descent has an implicit bias
towards flat minima. In deep neural network training, this mechanism serves to
screen out minima. However, the precise effect that this has on the trained
network is not yet fully understood. In this paper, we characterize the flat
minima in linear neural networks trained with a quadratic loss. First, we show
that linear ResNets with zero initialization necessarily converge to the
flattest of all minima. We then prove that these minima correspond to nearly
balanced networks whereby the gain from the input to any intermediate
representation does not change drastically from one layer to the next. Finally,
we show that consecutive layers in flat minima solutions are coupled. That is,
one of the left singular vectors of each weight matrix, equals one of the right
singular vectors of the next matrix. This forms a distinct path from input to
output, that, as we show, is dedicated to the signal that experiences the
largest gain end-to-end. Experiments indicate that these properties are
characteristic of both linear and nonlinear models trained in practice.

研究表明，随机梯度下降有一个偏好于平滑最小值的隐含偏差。 本文研究发现，在具有二次损失的线性神经网络训练中，线性 ResNets 的零初始化必然收敛于所有最小值中最平滑的最小值，这些最小值对应着接近平衡网络。另外，相邻层的权重矩阵在平坦的极小值解中相互耦合，形成了从输入到输出的明显路径，该路径只用于体验端到端最大增益的信号。