The success of deep learning, a brain-inspired form of AI, has sparked interest in understanding how the brain could similarly learn across multiple layers of neurons. However, the majority of biologically-plausible learning algorithms have not yet reached the performance of backpropagation (BP), nor are they built on strong theoretical foundations. Here, we analyze target propagation (TP), a popular but not yet fully understood alternative to BP, from the standpoint of mathematical optimization. Our theory shows that TP is closely related to Gauss-Newton optimization and thus substantially differs from BP. Furthermore, our analysis reveals a fundamental limitation of difference target propagation (DTP), a well-known variant of TP, in the realistic scenario of non-invertible neural networks. We provide a first solution to this problem through a novel reconstruction loss that improves feedback weight training, while simultaneously introducing architectural flexibility by allowing for direct feedback connections from the output to each hidden layer. Our theory is corroborated by experimental results that show significant improvements in performance and in the alignment of forward weight updates with loss gradients, compared to DTP.

本研究中，我们分析了一种叫做目标传播（TP）的替代反向传播（BP）的方法，从数学优化的角度进行了研究，并发现了它的基本限制。我们提出了一个创新性的重构损失来改善反馈权重训练，并通过允许直接反馈连接从输出到每个隐藏层来引入架构灵活性。实验结果表明，与差分目标传播（DTP）相比，性能得到了显著提高，并且前向权重更新与损失梯度的对准得到了改善。

目标传播的理论框架