We perform a careful, thorough, and large scale empirical study of the
correspondence between wide neural networks and kernel methods. By doing so, we
resolve a variety of open questions related to the study of infinitely wide
neural networks. Our experimental results include: kernel methods outperform
fully-connected finite-width networks, but underperform convolutional finite
width networks; neural network Gaussian process (NNGP) kernels frequently
outperform neural tangent (NT) kernels; centered and ensembled finite networks
have reduced posterior variance and behave more similarly to infinite networks;
weight decay and the use of a large learning rate break the correspondence
between finite and infinite networks; the NTK parameterization outperforms the
standard parameterization for finite width networks; diagonal regularization of
kernels acts similarly to early stopping; floating point precision limits
kernel performance beyond a critical dataset size; regularized ZCA whitening
improves accuracy; finite network performance depends non-monotonically on
width in ways not captured by double descent phenomena; equivariance of CNNs is
only beneficial for narrow networks far from the kernel regime. Our experiments
additionally motivate an improved layer-wise scaling for weight decay which
improves generalization in finite-width networks. Finally, we develop improved
best practices for using NNGP and NT kernels for prediction, including a novel
ensembling technique. Using these best practices we achieve state-of-the-art
results on CIFAR-10 classification for kernels corresponding to each
architecture class we consider.

通过实验证明了核方法优于限制宽度的全连接神经网络，并且证实 NNGP 内核经常优于 NT 内核，但它们的性能都受到正则化的影响。此外，作者提出了使用 NNGP 和 NT 内核进行预测的最佳实践方法，并在 CIFAR-10 分类任务中取得了最优结果。