Backpropagation (BP) has been a successful optimization technique for deep
learning models. However, its limitations, such as backward- and
update-locking, and its biological implausibility, hinder the concurrent
updating of layers and do not mimic the local learning processes observed in
the human brain. To address these issues, recent research has suggested using
local error signals to asynchronously train network blocks. However, this
approach often involves extensive trial-and-error iterations to determine the
best configuration for local training. This includes decisions on how to
decouple network blocks and which auxiliary networks to use for each block. In
our work, we introduce a novel BP-free approach: a block-wise BP-free (BWBPF)
neural network that leverages local error signals to optimize distinct
sub-neural networks separately, where the global loss is only responsible for
updating the output layer. The local error signals used in the BP-free model
can be computed in parallel, enabling a potential speed-up in the weight update
process through parallel implementation. Our experimental results consistently
show that this approach can identify transferable decoupled architectures for
VGG and ResNet variations, outperforming models trained with end-to-end
backpropagation and other state-of-the-art block-wise learning techniques on
datasets such as CIFAR-10 and Tiny-ImageNet. The code is released at
this https URL

通过引入一种新的无反向传播的方法，本研究提出了一个基于局部误差信号的块状神经网络，可以分别优化不同的子神经网络，并通过并行计算局部误差信号来加速权重更新过程，取得了比使用端到端反向传播和其他最先进的块状学习技术更好的性能，特别是在 CIFAR-10 和 Tiny-ImageNet 数据集上。

解锁深度学习：神经网络并行分块训练的 BP-Free 方法

Unlocking Deep Learning: A BP-Free Approach for Parallel Block-Wise  Training of Neural Networks

The collective behavior of a network with heterogeneous, resource-limited
information processing units (e.g., group of fish, flock of birds, or network
of neurons) demonstrates high self-organization and complexity. These emergent
properties arise from simple interaction rules where certain individuals can
exhibit leadership-like behavior and influence the collective activity of the
group. Motivated by the intricacy of these collectives, we propose a neural
network (NN) architecture inspired by the rules observed in nature's collective
ensembles. This NN structure contains workers that encompass one or more
information processing units (e.g., neurons, filters, layers, or blocks of
layers). Workers are either leaders or followers, and we train a
leader-follower neural network (LFNN) by leveraging local error signals and
optionally incorporating backpropagation (BP) and global loss. We investigate
worker behavior and evaluate LFNNs through extensive experimentation. Our LFNNs
trained with local error signals achieve significantly lower error rates than
previous BP-free algorithms on MNIST and CIFAR-10 and even surpass BP-enabled
baselines. In the case of ImageNet, our LFNN-l demonstrates superior
scalability and outperforms previous BP-free algorithms by a significant
margin.

提出了一种灵感来源于自然界集合组织规则的神经网络架构，该架构包含领导者和追随者，并通过利用局部误差信号和全局损失进行训练，实现了比以往基于反向传播算法的 MNIST 和 CIFAR-10 算法具有显著更低的错误率，甚至在 ImageNet 的情况下，其 LFNN-l 表现出更好的可伸缩性，且优于以往基于反向传播算法的算法。