Recent works have revealed that infinitely-wide feed-forward or recurrent
neural networks of any architecture correspond to Gaussian processes referred
to as Neural Network Gaussian Processes (NNGPs). While these works have
extended the class of neural networks converging to Gaussian processes
significantly, however, there has been little focus on broadening the class of
stochastic processes that such neural networks converge to. In this work,
inspired by the scale mixture of Gaussian random variables, we propose the
scale mixture of NNGPs for which we introduce a prior distribution on the scale
of the last-layer parameters. We show that simply introducing a scale prior on
the last-layer parameters can turn infinitely-wide neural networks of any
architecture into a richer class of stochastic processes. With certain scale
priors, we obtain heavy-tailed stochastic processes, and in the case of inverse
gamma priors, we recover Student's $t$ processes. We further analyze the
distributions of the neural networks initialized with our prior setting and
trained with gradient descents and obtain similar results as for NNGPs. We
present a practical posterior-inference algorithm for the scale mixture of
NNGPs and empirically demonstrate its usefulness on regression and
classification tasks. In particular, we show that in both tasks, the
heavy-tailed stochastic processes obtained from our framework are robust to
out-of-distribution data.

本文提出了一种基于高斯随机变量尺度混合的 NNGPs 模型，并利用先验分布在最后一层参数上引入尺度先验，使得任何架构的无穷宽神经网络都能转化为一种更丰富的随机过程，通过实验进一步证明了该模型在回归和分类任务方面的可行性和鲁棒性。