Understanding the impact of data structure on the computational tractability
of learning is a key challenge for the theory of neural networks. Many
theoretical works do not explicitly model training data, or assume that inputs
are drawn component-wise independently from some simple probability
distribution. Here, we go beyond this simple paradigm by studying the
performance of neural networks trained on data drawn from pre-trained
generative models. This is possible due to a Gaussian equivalence stating that
the key metrics of interest, such as the training and test errors, can be fully
captured by an appropriately chosen Gaussian model. We provide three strands of
rigorous, analytical and numerical evidence corroborating this equivalence.
First, we establish rigorous conditions for the Gaussian equivalence to hold in
the case of single-layer generative models, as well as deterministic rates for
convergence in distribution. Second, we leverage this equivalence to derive a
closed set of equations describing the generalisation performance of two widely
studied machine learning problems: two-layer neural networks trained using
one-pass stochastic gradient descent, and full-batch pre-learned features or
kernel methods. Finally, we perform experiments demonstrating how our theory
applies to deep, pre-trained generative models. These results open a viable
path to the theoretical study of machine learning models with realistic data.

通过研究神经网络在预训练生成模型产生的数据集上的训练和表现，探讨数据结构对学习的计算效率的影响，证明了高斯等价性可以使用一个适当选择的高斯模型来完全捕获感兴趣的关键性能指标。