We investigate how the final parameters found by stochastic gradient descent are influenced by over-parameterization. We generate families of models by increasing the number of channels in a base network, and then perform a large hyper-parameter search to study how the test error depends on learning rate, batch size, and network width. We find that the optimal SGD hyper-parameters are determined by a "normalized noise scale," which is a function of the batch size, learning rate, and initialization conditions. In the absence of batch normalization, the optimal normalized noise scale is directly proportional to width. Wider networks, with their higher optimal noise scale, also achieve higher test accuracy. These observations hold for MLPs, ConvNets, and ResNets, and for two different parameterization schemes ("Standard" and "NTK"). We observe a similar trend with batch normalization for ResNets. Surprisingly, since the largest stable learning rate is bounded, the largest batch size consistent with the optimal normalized noise scale decreases as the width increases.

通过超参寻优与一系列不同的神经网络模型，我们探究了过度参数化情况下随机梯度下降最终调整到的参数状态，并发现优化算法的最优超参数取决于一项被规范化的噪声参数，其与网络宽度等因素有关，这对各种类型的神经网络均成立，对于 ResNets 这样的网络同样存在相似的趋势。

网络宽度对随机梯度下降和泛化效果的影响：实证研究