We consider a large class of shallow neural networks with randomly
initialized parameters and rectified linear unit activation functions. We prove
that these random neural networks are well-defined non-Gaussian processes. As a
by-product, we demonstrate that these networks are solutions to stochastic
differential equations driven by impulsive white noise (combinations of random
Dirac measures). These processes are parameterized by the law of the weights
and biases as well as the density of activation thresholds in each bounded
region of the input domain. We prove that these processes are isotropic and
wide-sense self-similar with Hurst exponent $3/2$. We also derive a remarkably
simple closed-form expression for their autocovariance function. Our results
are fundamentally different from prior work in that we consider a
non-asymptotic viewpoint: The number of neurons in each bounded region of the
input domain (i.e., the width) is itself a random variable with a Poisson law
with mean proportional to the density parameter. Finally, we show that, under
suitable hypotheses, as the expected width tends to infinity, these processes
can converge in law not only to Gaussian processes, but also to non-Gaussian
processes depending on the law of the weights. Our asymptotic results provide a
new take on several classical results (wide networks converge to Gaussian
processes) as well as some new ones (wide networks can converge to non-Gaussian
processes).

我们研究了具有随机初始化参数和修正线性单元激活函数的大类浅层神经网络，并证明了这些随机神经网络是良定义的非高斯过程，由脉冲白噪声（随机狄拉克测度的组合）驱动的随机微分方程的解。这些过程由权重和偏置的分布以及输入域中每个有界区域中激活阈值的密度所参数化。我们证明这些过程是等向的，同时具有 Hurst 指数为 3/2 的广义自相似性，并导出了它们的自协方差函数的一个非常简单的闭式表达式。我们的结果从非渐近的视角来看与先前的工作有本质不同：输入域中每个有界区域（即宽度）的神经元数量本身是一个具有泊松分布的随机变量，并且其均值与密度参数成比例。最后，我们证明在适当的假设下，当期望宽度趋于无穷大时，这些过程除了能收敛到高斯过程外，还能收敛到依赖于权重分布的非高斯过程。我们的渐近结果提供了一种新的方法来看待几个经典结果（宽网络收敛到高斯过程）以及一些新结果（宽网络可以收敛到非高斯过程）。

随机 ReLU 神经网络作为非高斯过程

Random ReLU Neural Networks as Non-Gaussian Processes

Gaussian processes are ubiquitous in nature and engineering. A case in point
is a class of neural networks in the infinite-width limit, whose priors
correspond to Gaussian processes. Here we perturbatively extend this
correspondence to finite-width neural networks, yielding non-Gaussian processes
as priors. The methodology developed herein allows us to track the flow of
preactivation distributions by progressively integrating out random variables
from lower to higher layers, reminiscent of renormalization-group flow. We
further develop a perturbative procedure to perform Bayesian inference with
weakly non-Gaussian priors.

本文研究基于高斯过程的无限宽神经网络的有限宽度情况，通过积分消除随机变量实现层间激活分布的跟踪，推导出非高斯过程作为其先验，并发展了一种弱非高斯先验下的贝叶斯推断扰动程序。