We show at a physics level of rigor that Bayesian inference with a fully
connected neural network and a shaped nonlinearity of the form $\phi(t) = t +
\psi t^3/L$ is (perturbatively) solvable in the regime where the number of
training datapoints $P$ , the input dimension $N_0$, the network layer widths
$N$, and the network depth $L$ are simultaneously large. Our results hold with
weak assumptions on the data; the main constraint is that $P < N_0$. We provide
techniques to compute the model evidence and posterior to arbitrary order in
$1/N$ and at arbitrary temperature. We report the following results from the
first-order computation:
1. When the width $N$ is much larger than the depth $L$ and training set size
$P$, neural network Bayesian inference coincides with Bayesian inference using
a kernel. The value of $\psi$ determines the curvature of a sphere, hyperbola,
or plane into which the training data is implicitly embedded under the feature
map.
2. When $LP/N$ is a small constant, neural network Bayesian inference departs
from the kernel regime. At zero temperature, neural network Bayesian inference
is equivalent to Bayesian inference using a data-dependent kernel, and $LP/N$
serves as an effective depth that controls the extent of feature learning.
3. In the restricted case of deep linear networks ($\psi=0$) and noisy data,
we show a simple data model for which evidence and generalization error are
optimal at zero temperature. As $LP/N$ increases, both evidence and
generalization further improve, demonstrating the benefit of depth in benign
overfitting.

在大量训练数据、输入维度、网络层宽度和网络深度同时很大的情况下，我们展示了贝叶斯推断与全连接神经网络和形状非线性的关系，并提供了计算模型证据和后验的技术，结果表明神经网络贝叶斯推断与使用核函数的贝叶斯推断相一致，当网络层宽度大于深度和训练集大小时，神经网络贝叶斯推断的深度是一个有效的参数。