We investigate the stationary (late-time) training regime of single- and
two-layer linear neural networks within the continuum limit of stochastic
gradient descent (SGD) for synthetic Gaussian data. In the case of a
single-layer network in the weakly oversampled regime, the spectrum of the
noise covariance matrix deviates notably from the Hessian, which can be
attributed to the broken detailed balance of SGD dynamics. The weight
fluctuations are in this case generally anisotropic, but experience an
isotropic loss. For a two-layer network, we obtain the stochastic dynamics of
the weights in each layer and analyze the associated stationary covariances. We
identify the inter-layer coupling as a new source of anisotropy for the weight
fluctuations. In contrast to the single-layer case, the weight fluctuations
experience an anisotropic loss, the flatness of which is inversely related to
the fluctuation variance. We thereby provide an analytical derivation of the
recently observed inverse variance-flatness relation in a deep linear network
model.

我们研究了合成高斯数据中单层和双层线性神经网络在随机梯度下降（SGD）的持续（末期）训练阶段。我们发现，在过度采样不足的弱过采样范围内，单层网络的噪声协方差矩阵的谱明显偏离 Hessian，可归因于 SGD 动力学的详细平衡破坏。在这种情况下，权重波动一般是各向异性的，但经历各向同性的损失。对于双层网络，我们获得了各层权重的随机动力学，并分析了相关的稳定的协方差。我们将层间耦合确定为权重波动的新源头。与单层情况不同，权重波动经历了各向异性的损失，其平坦度与波动方差成反比。因此，我们在深度线性网络模型中提供了最近观察到的逆方差 - 平坦度关系的分析推导。