Artificial neural networks have revolutionized machine learning in recent
years, but a complete theoretical framework for their learning process is still
lacking. Substantial progress has been made for infinitely wide networks. In
this regime, two disparate theoretical frameworks have been used, in which the
network's output is described using kernels: one framework is based on the
Neural Tangent Kernel (NTK) which assumes linearized gradient descent dynamics,
while the Neural Network Gaussian Process (NNGP) kernel assumes a Bayesian
framework. However, the relation between these two frameworks has remained
elusive. This work unifies these two distinct theories using a Markov proximal
learning model for learning dynamics in an ensemble of randomly initialized
infinitely wide deep networks. We derive an exact analytical expression for the
network input-output function during and after learning, and introduce a new
time-dependent Neural Dynamical Kernel (NDK) from which both NTK and NNGP
kernels can be derived. We identify two learning phases characterized by
different time scales: gradient-driven and diffusive learning. In the initial
gradient-driven learning phase, the dynamics is dominated by deterministic
gradient descent, and is described by the NTK theory. This phase is followed by
the diffusive learning stage, during which the network parameters sample the
solution space, ultimately approaching the equilibrium distribution
corresponding to NNGP. Combined with numerical evaluations on synthetic and
benchmark datasets, we provide novel insights into the different roles of
initialization, regularization, and network depth, as well as phenomena such as
early stopping and representational drift. This work closes the gap between the
NTK and NNGP theories, providing a comprehensive framework for understanding
the learning process of deep neural networks in the infinite width limit.

通过使用具有随机初始化的无限宽度深度网络集合的马尔可夫接近学习模型，结合数值评估来合并和统一神经切向核（NTK）和神经网络高斯过程（NNGP）理论，并提供对机器学习中深度神经网络学习过程的全面理解。

连接 NTK 和 NNGP：核区域中神经网络学习动力学的统一理论框架

Connecting NTK and NNGP: A Unified Theoretical Framework for Neural  Network Learning Dynamics in the Kernel Regime

Infinite width limit has shed light on generalization and optimization
aspects of deep learning by establishing connections between neural networks
and kernel methods. Despite their importance, the utility of these kernel
methods was limited in large-scale learning settings due to their
(super-)quadratic runtime and memory complexities. Moreover, most prior works
on neural kernels have focused on the ReLU activation, mainly due to its
popularity but also due to the difficulty of computing such kernels for general
activations. In this work, we overcome such difficulties by providing methods
to work with general activations. First, we compile and expand the list of
activation functions admitting exact dual activation expressions to compute
neural kernels. When the exact computation is unknown, we present methods to
effectively approximate them. We propose a fast sketching method that
approximates any multi-layered Neural Network Gaussian Process (NNGP) kernel
and Neural Tangent Kernel (NTK) matrices for a wide range of activation
functions, going beyond the commonly analyzed ReLU activation. This is done by
showing how to approximate the neural kernels using the truncated Hermite
expansion of any desired activation functions. While most prior works require
data points on the unit sphere, our methods do not suffer from such limitations
and are applicable to any dataset of points in $\mathbb{R}^d$. Furthermore, we
provide a subspace embedding for NNGP and NTK matrices with near input-sparsity
runtime and near-optimal target dimension which applies to any
\emph{homogeneous} dual activation functions with rapidly convergent Taylor
expansion. Empirically, with respect to exact convolutional NTK (CNTK)
computation, our method achieves $106\times$ speedup for approximate CNTK of a
5-layer Myrtle network on CIFAR-10 dataset.

本文提出了一种基于截断埃尔米特函数的方法，用于近似计算任何多层神经网络的高斯过程核（NNGP）和神经切向核（NTK）矩阵，同时克服了其他方法中数据点必须在单位球上的限制，可适用于任何 $R^d$ 空间中的点集。实验证明，相对于精确的卷积神经切向核计算，该方法在 CIFAR-10 数据集上针对具有 5 层结构的 Myrtle 网络的近似计算中实现了 106 倍的加速。