The development of biologically-plausible learning algorithms is important
for understanding learning in the brain, but most of them fail to scale-up to
real-world tasks, limiting their potential as explanations for learning by real
brains. As such, it is important to explore learning algorithms that come with
strong theoretical guarantees and can match the performance of backpropagation
(BP) on complex tasks. One such algorithm is Difference Target Propagation
(DTP), a biologically-plausible learning algorithm whose close relation with
Gauss-Newton (GN) optimization has been recently established. However, the
conditions under which this connection rigorously holds preclude layer-wise
training of the feedback pathway synaptic weights (which is more biologically
plausible). Moreover, good alignment between DTP weight updates and loss
gradients is only loosely guaranteed and under very specific conditions for the
architecture being trained. In this paper, we propose a novel feedback weight
training scheme that ensures both that DTP approximates BP and that layer-wise
feedback weight training can be restored without sacrificing any theoretical
guarantees. Our theory is corroborated by experimental results and we report
the best performance ever achieved by DTP on CIFAR-10 and ImageNet 32$\times$32

本文提出了一种新型的反馈权重训练方案，确保 DTP 逼近 BP，并且可以恢复逐层反馈权重训练，同时不会牺牲任何理论保证，实验证实了该理论，并报告了 DTP 在 CIFAR-10 和 ImageNet 32×32 上取得的最佳表现。