This paper presents two models of neural-networks and their training
applicable to neural networks of arbitrary width, depth and topology, assuming
only finite-energy neural activations; and a novel representor theory for
neural networks in terms of a matrix-valued kernel. The first model is exact
(un-approximated) and global, casting the neural network as an elements in a
reproducing kernel Banach space (RKBS); we use this model to provide tight
bounds on Rademacher complexity. The second model is exact and local, casting
the change in neural network function resulting from a bounded change in
weights and biases (ie. a training step) in reproducing kernel Hilbert space
(RKHS) in terms of a local-intrinsic neural kernel (LiNK). This local model
provides insight into model adaptation through tight bounds on Rademacher
complexity of network adaptation. We also prove that the neural tangent kernel
(NTK) is a first-order approximation of the LiNK kernel. Finally, and noting
that the LiNK does not provide a representor theory for technical reasons, we
present an exact novel representor theory for layer-wise neural network
training with unregularized gradient descent in terms of a local-extrinsic
neural kernel (LeNK). This representor theory gives insight into the role of
higher-order statistics in neural network training and the effect of kernel
evolution in neural-network kernel models. Throughout the paper (a) feedforward
ReLU networks and (b) residual networks (ResNet) are used as illustrative
examples.

这篇论文提出了两种适用于任意宽度、深度和拓扑结构的神经网络的模型及其训练方法，假设神经激活仅存在有限能量，并提出了一种基于矩阵核的神经网络的新颖表达器理论。其中，第一种模型是精确的且全局的模型，将神经网络看作是再生核 Banach 空间中的元素；第二种模型是精确的且局部的模型，将神经网络函数的变化建模为权重和偏差（即训练步骤）在再生核 Hilbert 空间中的局部内在神经核。该局部模型通过对网络自适应的 Rademacher 复杂度提供洞察力。此外，还证明了神经切向核 (NTK) 是局部内在神经核的一阶近似。最后，针对技术原因，本论文提出了一个准确的新颖表达器理论，用于层间神经网络在未正则化梯度下降的情况下进行训练，该理论基于局部外在神经核 (LeNK)。该表达器理论揭示了神经网络训练中高阶统计量的作用以及核演化对神经网络核模型的影响。在整篇论文中，使用前馈 ReLU 网络和残差网络 (ResNet) 作为说明性例子。