The largest eigenvalue of the Hessian, or sharpness, of neural networks is a
key quantity to understand their optimization dynamics. In this paper, we study
the sharpness of deep linear networks for overdetermined univariate regression.
Minimizers can have arbitrarily large sharpness, but not an arbitrarily small
one. Indeed, we show a lower bound on the sharpness of minimizers, which grows
linearly with depth. We then study the properties of the minimizer found by
gradient flow, which is the limit of gradient descent with vanishing learning
rate. We show an implicit regularization towards flat minima: the sharpness of
the minimizer is no more than a constant times the lower bound. The constant
depends on the condition number of the data covariance matrix, but not on width
or depth. This result is proven both for a small-scale initialization and a
residual initialization. Results of independent interest are shown in both
cases. For small-scale initialization, we show that the learned weight matrices
are approximately rank-one and that their singular vectors align. For residual
initialization, convergence of the gradient flow for a Gaussian initialization
of the residual network is proven. Numerical experiments illustrate our results
and connect them to gradient descent with non-vanishing learning rate.

神经网络的海森矩阵的最大特征值（或清晰度）是理解其优化动态的关键量。本文研究超定单变量回归的深度线性网络的清晰度。虽然最小化器的清晰度可以任意大，但不可以任意小。事实上，我们证明了最小化器清晰度的下界与深度成线性增长。然后我们研究了梯度流找到的最小化器的性质，这是梯度下降的极限情况，学习率趋于零。我们证明了对于平坦最小值的隐式正则化：最小化器的清晰度不超过下界的一个常数倍。该常数取决于数据协方差矩阵的条件数，而不取决于宽度或深度。我们分别证明了小尺度初始化和残差初始化的结果。对于小尺度初始化，我们证明了所学权重矩阵近似为秩一及其奇异向量对齐。对于残差初始化，我们证明了高斯初始化的残差网络的梯度流的收敛性。数值实验验证了我们的结果，并将其与非零学习率的梯度下降联系在一起。