Forward gradient learning computes a noisy directional gradient and is a
biologically plausible alternative to backprop for learning deep neural
networks. However, the standard forward gradient algorithm, when applied
naively, suffers from high variance when the number of parameters to be learned
is large. In this paper, we propose a series of architectural and algorithmic
modifications that together make forward gradient learning practical for
standard deep learning benchmark tasks. We show that it is possible to
substantially reduce the variance of the forward gradient estimator by applying
perturbations to activations rather than weights. We further improve the
scalability of forward gradient by introducing a large number of local greedy
loss functions, each of which involves only a small number of learnable
parameters, and a new MLPMixer-inspired architecture, LocalMixer, that is more
suitable for local learning. Our approach matches backprop on MNIST and
CIFAR-10 and significantly outperforms previously proposed backprop-free
algorithms on ImageNet.

本文介绍了一种改进的前向梯度学习算法，通过应用激活项的扰动和引入大量的局部贪婪损失函数以及新的局部学习结构 LocalMixer，可以显著降低梯度估计的方差，实现在 MNIST、CIFAR-10 和 ImageNet 数据集上与反向传播算法相媲美的结果。