The ubiquitous backpropagation algorithm requires sequential updates across
blocks of a network, introducing a locking problem. Moreover, backpropagation
relies on the transpose of weight matrices to calculate updates, introducing a
weight transport problem across blocks. Both these issues prevent efficient
parallelisation and horizontal scaling of models across devices. We propose a
new method that introduces a twin network that propagates information backwards
from the targets to the input to provide auxiliary local losses. Forward and
backward propagation can work in parallel and with different sets of weights,
addressing the problems of weight transport and locking. Our approach derives
from a statistical interpretation of end-to-end training which treats
activations of network layers as parameters of probability distributions. The
resulting learning framework uses these parameters locally to assess the
matching between forward and backward information. Error backpropagation is
then performed locally within each block, leading to `block-local' learning.
Several previously proposed alternatives to error backpropagation emerge as
special cases of our model. We present results on various tasks and
architectures, including transformers, demonstrating state-of-the-art
performance using block-local learning. These results provide a new principled
framework to train very large networks in a distributed setting and can also be
applied in neuromorphic systems.

通过引入双网络的反向传播方法和将网络中的层激活视作概率分布的参数，本文提出了一种解决反向传播中锁死和权重传输问题的新方法，从而实现对大型网络的分布式高效训练。相应的实验结果表明了其在多种任务和结构上的优越表现。