The ubiquitous backpropagation algorithm requires sequential updates across
blocks of a network, introducing a locking problem. Moreover, backpropagation
relies on the transpose of weight matrices to calculate updates, introducing a
weight transport problem across blocks. Both these issues prevent efficient
parallelisation and horizontal scaling of models across devices. We propose a
new method that introduces a twin network that propagates information backwards
from the targets to the input to provide auxiliary local losses. Forward and
backward propagation can work in parallel and with different sets of weights,
addressing the problems of weight transport and locking. Our approach derives
from a statistical interpretation of end-to-end training which treats
activations of network layers as parameters of probability distributions. The
resulting learning framework uses these parameters locally to assess the
matching between forward and backward information. Error backpropagation is
then performed locally within each block, leading to `block-local' learning.
Several previously proposed alternatives to error backpropagation emerge as
special cases of our model. We present results on various tasks and
architectures, including transformers, demonstrating state-of-the-art
performance using block-local learning. These results provide a new principled
framework to train very large networks in a distributed setting and can also be
applied in neuromorphic systems.

通过引入双网络的反向传播方法和将网络中的层激活视作概率分布的参数，本文提出了一种解决反向传播中锁死和权重传输问题的新方法，从而实现对大型网络的分布式高效训练。相应的实验结果表明了其在多种任务和结构上的优越表现。

基于概率潜在表示的块局部学习

Block-local learning with probabilistic latent representations

In artificial neural networks trained with gradient descent, the weights used
for processing stimuli are also used during backward passes to calculate
gradients. For the real brain to approximate gradients, gradient information
would have to be propagated separately, such that one set of synaptic weights
is used for processing and another set is used for backward passes. This
produces the so-called "weight transport problem" for biological models of
learning, where the backward weights used to calculate gradients need to mirror
the forward weights used to process stimuli. This weight transport problem has
been considered so hard that popular proposals for biological learning assume
that the backward weights are simply random, as in the feedback alignment
algorithm. However, such random weights do not appear to work well for large
networks. Here we show how the discontinuity introduced in a spiking system can
lead to a solution to this problem. The resulting algorithm is a special case
of an estimator used for causal inference in econometrics, regression
discontinuity design. We show empirically that this algorithm rapidly makes the
backward weights approximate the forward weights. As the backward weights
become correct, this improves learning performance over feedback alignment on
tasks such as Fashion-MNIST, SVHN, CIFAR-10 and VOC. Our results demonstrate
that a simple learning rule in a spiking network can allow neurons to produce
the right backward connections and thus solve the weight transport problem.

通过引入脉冲系统处理信息，使得反向传导和前向传输的神经突触权重不相同，解决了生物模型中学习时的权重传递问题，提高了学习表现。

基于脉冲的因果推理用于权值对齐

Spike-based causal inference for weight alignment

The backpropagation (BP) algorithm is often thought to be biologically
implausible in the brain. One of the main reasons is that BP requires symmetric
weight matrices in the feedforward and feedback pathways. To address this
"weight transport problem" (Grossberg, 1987), two more biologically plausible
algorithms, proposed by Liao et al. (2016) and Lillicrap et al. (2016), relax
BP's weight symmetry requirements and demonstrate comparable learning
capabilities to that of BP on small datasets. However, a recent study by
Bartunov et al. (2018) evaluate variants of target-propagation (TP) and
feedback alignment (FA) on MINIST, CIFAR, and ImageNet datasets, and find that
although many of the proposed algorithms perform well on MNIST and CIFAR, they
perform significantly worse than BP on ImageNet. Here, we additionally evaluate
the sign-symmetry algorithm (Liao et al., 2016), which differs from both BP and
FA in that the feedback and feedforward weights share signs but not magnitudes.
We examine the performance of sign-symmetry and feedback alignment on ImageNet
and MS COCO datasets using different network architectures (ResNet-18 and
AlexNet for ImageNet, RetinaNet for MS COCO). Surprisingly, networks trained
with sign-symmetry can attain classification performance approaching that of
BP-trained networks. These results complement the study by Bartunov et al.
(2018), and establish a new benchmark for future biologically plausible
learning algorithms on more difficult datasets and more complex architectures.

研究了为了解决反向传播算法对称性问题的不同技术及其对不同数据集和网络结构的适用性，其中 sign-symmetry 算法展现出了逼近 BP 算法的分类性能。