A commonly cited inefficiency of neural network training by back-propagation
is the update locking problem: each layer must wait for the signal to propagate
through the full network before updating. Several alternatives that can
alleviate this issue have been proposed. In this context, we consider a
simpler, but more effective, substitute that uses minimal feedback, which we
call Decoupled Greedy Learning (DGL). It is based on a greedy relaxation of the
joint training objective, recently shown to be effective in the context of
Convolutional Neural Networks (CNNs) on large-scale image classification. We
consider an optimization of this objective that permits us to decouple the
layer training, allowing for layers or modules in networks to be trained with a
potentially linear parallelization in layers. With the use of a replay buffer
we show this approach can be extended to asynchronous settings, where modules
can operate with possibly large communication delays. We show theoretically and
empirically that this approach converges. Then, we empirically find that it can
lead to better generalization than sequential greedy optimization. We
demonstrate the effectiveness of DGL against alternative approaches on the
CIFAR-10 dataset and on the large-scale ImageNet dataset.

本文提出了解决反向传播过程中每层必须等待信号传播整个网络才能更新的锁定问题的替代方案 Decoupled Greedy Learning，通过使用最小反馈进行网络训练，并探讨了该方案的优化方法，可以在线性层级并行化的情况下独立地训练网络层或模块，本方案能够在异步环境下运行，并且具有更好的泛化性能，实验证明其对于 CIFAR-10 数据集和大规模 ImageNet 数据集的分类效果优于其他方案。