There has recently been much work on the "wide limit" of neural networks,
where Bayesian neural networks (BNNs) are shown to converge to a Gaussian
process (GP) as all hidden layers are sent to infinite width. However, these
results do not apply to architectures that require one or more of the hidden
layers to remain narrow. In this paper, we consider the wide limit of BNNs
where some hidden layers, called "bottlenecks", are held at finite width. The
result is a composition of GPs that we term a "bottleneck neural network
Gaussian process" (bottleneck NNGP). Although intuitive, the subtlety of the
proof is in showing that the wide limit of a composition of networks is in fact
the composition of the limiting GPs. We also analyze theoretically a
single-bottleneck NNGP, finding that the bottleneck induces dependence between
the outputs of a multi-output network that persists through extreme
post-bottleneck depths, and prevents the kernel of the network from losing
discriminative power at extreme post-bottleneck depths.

本文探讨了存在 “瓶颈” 的 BNNs 的宽限制，其中一些隐藏层被保持在有限宽度，结果是一种高斯过程的组合，我们称之为 “瓶颈神经网络高斯过程”（bottleneck NNGP），并分析了单个瓶颈 NNGP 的理论，发现瓶颈在极端深度通过产生输出之间的相关性，并防止网络的内核在极端深度失去区分能力。