The predictions of wide Bayesian neural networks are described by a Gaussian
process, known as the Neural Network Gaussian Process (NNGP). Analytic forms
for NNGP kernels are known for many models, but computing the exact kernel for
convolutional architectures is prohibitively expensive. One can obtain
effective approximations of these kernels through Monte-Carlo estimation using
finite networks at initialization. Monte-Carlo NNGP inference is
orders-of-magnitude cheaper in FLOPs compared to gradient descent training when
the dataset size is small. Since NNGP inference provides a cheap measure of
performance of a network architecture, we investigate its potential as a signal
for neural architecture search (NAS). We compute the NNGP performance of
approximately 423k networks in the NAS-bench 101 dataset on CIFAR-10 and
compare its utility against conventional performance measures obtained by
shortened gradient-based training. We carry out a similar analysis on 10k
randomly sampled networks in the mobile neural architecture search (MNAS) space
for ImageNet. We discover comparative advantages of NNGP-based metrics, and
discuss potential applications. In particular, we propose that NNGP performance
is an inexpensive signal independent of metrics obtained from training that can
either be used for reducing big search spaces, or improving training-based
performance measures.

本文研究 Bayesian 神经网络，使用 Monte-Carlo NNGP 推断方法得到 NNGP 性能，探讨其在神经架构搜索中的应用，发现 NNGP 性能具有成本低廉、独立于训练度量且可以用于缩小大搜索空间或提高基于训练的性能度量的优点。