Understanding the reasons for the success of deep neural networks trained
using stochastic gradient-based methods is a key open problem for the nascent
theory of deep learning. The types of data where these networks are most
successful, such as images or sequences of speech, are characterised by
intricate correlations. Yet, most theoretical work on neural networks does not
explicitly model training data, or assumes that elements of each data sample
are drawn independently from some factorised probability distribution. These
approaches are thus by construction blind to the correlation structure of
real-world data sets and their impact on learning in neural networks. Here, we
introduce a generative model for structured data sets that we call the hidden
manifold model (HMM). The idea is to construct high-dimensional inputs that lie
on a lower-dimensional manifold, with labels that depend only on their position
within this manifold, akin to a single layer decoder or generator in a
generative adversarial network. We demonstrate that learning of the hidden
manifold model is amenable to an analytical treatment by proving a "Gaussian
Equivalence Property" (GEP), and we use the GEP to show how the dynamics of
two-layer neural networks trained using one-pass stochastic gradient descent is
captured by a set of integro-differential equations that track the performance
of the network at all times. This permits us to analyse in detail how a neural
network learns functions of increasing complexity during training, how its
performance depends on its size and how it is impacted by parameters such as
the learning rate or the dimension of the hidden manifold.

本文提出了一个名为 “隐藏流形模型” 的生成模型，证明了随机梯度下降算法训练的两层神经网络表现的动态可以通过 Integro-differential 方程组进行跟踪，同时分析了神经网络训练过程中神经网络学习增加复杂度和提高性能的方式以及受其大小、学习率和隐藏流形维度等参数的影响。