We carry out an information-theoretical analysis of a two-layer neural
network trained from input-output pairs generated by a teacher network with
matching architecture, in overparametrized regimes. Our results come in the
form of bounds relating i) the mutual information between training data and
network weights, or ii) the Bayes-optimal generalization error, to the same
quantities but for a simpler (generalized) linear model for which explicit
expressions are rigorously known. Our bounds, which are expressed in terms of
the number of training samples, input dimension and number of hidden units,
thus yield fundamental performance limits for any neural network (and actually
any learning procedure) trained from limited data generated according to our
two-layer teacher neural network model. The proof relies on rigorous tools from
spin glasses and is guided by ``Gaussian equivalence principles'' lying at the
core of numerous recent analyses of neural networks. With respect to the
existing literature, which is either non-rigorous or restricted to the case of
the learning of the readout weights only, our results are information-theoretic
(i.e. are not specific to any learning algorithm) and, importantly, cover a
setting where all the network parameters are trained.

对使用两层神经网络进行信息理论分析的研究，研究了限制数据和模型的情况下神经网络的性能极限，结果表明这个性能极限取决于训练数据的数量、输入维度和隐藏单元的数量