Past decades have witnessed a great interest in the distinction and
connection between neural network learning and kernel learning. Recent
advancements have made theoretical progress in connecting infinite-wide neural
networks and Gaussian processes. Two predominant approaches have emerged: the
Neural Network Gaussian Process (NNGP) and the Neural Tangent Kernel (NTK). The
former, rooted in Bayesian inference, represents a zero-order kernel, while the
latter, grounded in the tangent space of gradient descents, is a first-order
kernel. In this paper, we present the Unified Neural Kernel (UNK), which
characterizes the learning dynamics of neural networks with gradient descents
and parameter initialization. The proposed UNK kernel maintains the limiting
properties of both NNGP and NTK, exhibiting behaviors akin to NTK with a finite
learning step and converging to NNGP as the learning step approaches infinity.
Besides, we also theoretically characterize the uniform tightness and learning
convergence of the UNK kernel, providing comprehensive insights into this
unified kernel. Experimental results underscore the effectiveness of our
proposed method.

该论文介绍了一种名为 Unified Neural Kernel (UNK) 的方法，用于描述神经网络的学习动态以及参数初始化。通过渐进学习步骤，UNK 核的行为呈现类似于 Neural Tangent Kernel (NTK)，而随着学习步骤接近无穷大，其收敛于 Neural Network Gaussian Process (NNGP)。此外，论文还对 UNK 核的均匀紧密性和学习收敛性进行了理论表征，并通过实验证明了该方法的有效性。

神经网络学习的统一核心

A Unified Kernel for Neural Network Learning

Bayesian inference and kernel methods are well established in machine
learning. The neural network Gaussian process in particular provides a concept
to investigate neural networks in the limit of infinitely wide hidden layers by
using kernel and inference methods. Here we build upon this limit and provide a
field-theoretic formalism which covers the generalization properties of
infinitely wide networks. We systematically compute generalization properties
of linear, non-linear, and deep non-linear networks for kernel matrices with
heterogeneous entries. In contrast to currently employed spectral methods we
derive the generalization properties from the statistical properties of the
input, elucidating the interplay of input dimensionality, size of the training
data set, and variability of the data. We show that data variability leads to a
non-Gaussian action reminiscent of a ($\varphi^3+\varphi^4$)-theory. Using our
formalism on a synthetic task and on MNIST we obtain a homogeneous kernel
matrix approximation for the learning curve as well as corrections due to data
variability which allow the estimation of the generalization properties and
exact results for the bounds of the learning curves in the case of infinitely
many training data points.

基于无限宽度神经网络的高斯过程，并结合内核和推理方法，构建了一个场论的形式体系，研究了无限宽度网络的泛化性质，并从输入数据的统计性质得到了泛化性质的提取。