There are currently two parameterizations used to derive fixed kernels
corresponding to infinite width neural networks, the NTK (Neural Tangent
Kernel) parameterization and the naive standard parameterization. However, the
extrapolation of both of these parameterizations to infinite width is
problematic. The standard parameterization leads to a divergent neural tangent
kernel while the NTK parameterization fails to capture crucial aspects of
finite width networks such as: the dependence of training dynamics on relative
layer widths, the relative training dynamics of weights and biases, and overall
learning rate scale. Here we propose an improved extrapolation of the standard
parameterization that preserves all of these properties as width is taken to
infinity and yields a well-defined neural tangent kernel. We show
experimentally that the resulting kernels typically achieve similar accuracy to
those resulting from an NTK parameterization, but with better correspondence to
the parameterization of typical finite width networks. Additionally, with
careful tuning of width parameters, the improved standard parameterization
kernels can outperform those stemming from an NTK parameterization. We release
code implementing this improved standard parameterization as part of the Neural
Tangents library at this https URL

本文提出了一种改进的标准参数化方法，可以正确地在无限宽度的情况下捕捉到有限宽度网络的训练动态，同时保持所有属性，包括神经切线核，实验表明这种参数化方法通常可以达到与 NTK 参数化类似的精度，但更好地对应典型有限宽度网络的参数化。