Recent works have examined how deep neural networks, which can solve a variety of difficult problems, incorporate the statistics of training data to achieve their success. However, existing results have been established only in limited settings. In this work, we derive the layerwise weight dynamics of infinite-width neural networks with nonlinear activations trained by gradient descent. We show theoretically that weight updates are aligned with input correlations from intermediate layers weighted by error, and demonstrate empirically that the result also holds in finite-width wide networks. The alignment result allows us to formulate backpropagation-free learning rules, named Align-zero and Align-ada, that theoretically achieve the same alignment as backpropagation. Finally, we test these learning rules on benchmark problems in feedforward and recurrent neural networks and demonstrate, in wide networks, comparable performance to backpropagation.

本文通过考虑神经元的宽度，利用神经切向核学习理论研究了神经网络在生物学中的可能模型，并提出了一种基于输入活动相关性的反向传播算法，该方法在低数据环境中表现出与反向传播相当的性能。

无需反向传播的宽神经网络训练方法：基于输入权重对齐的视角