Deep neural networks achieve stellar generalisation even when they have
enough parameters to easily fit all their training data. We study this
phenomenon by analysing the dynamics and the performance of over-parameterised
two-layer neural networks in the teacher-student setup, where one network, the
student, is trained on data generated by another network, called the teacher.
We show how the dynamics of stochastic gradient descent (SGD) is captured by a
set of differential equations and prove that this description is asymptotically
exact in the limit of large inputs. Using this framework, we calculate the
final generalisation error of student networks that have more parameters than
their teachers. We find that the final generalisation error of the student
increases with network size when training only the first layer, but stays
constant or even decreases with size when training both layers. We show that
these different behaviours have their root in the different solutions SGD finds
for different activation functions. Our results indicate that achieving good
generalisation in neural networks goes beyond the properties of SGD alone and
depends on the interplay of at least the algorithm, the model architecture, and
the data set.

在大规模数据训练下，研究了超参数化两层神经网络在教师 - 学生的设置下的动态和表现，证明了 SGD 的一组微分方程可以捕捉到动态，同时揭示了不同激活函数找到的解决方案的不同表现，并发现了对于神经网络的良好泛化，不仅取决于 SGD 的性质，还取决于算法，模型架构和数据集的相互作用。