Target Propagation (TP) is a biologically more plausible algorithm than the
error backpropagation (BP) to train deep networks, and improving practicality
of TP is an open issue. TP methods require the feedforward and feedback
networks to form layer-wise autoencoders for propagating the target values
generated at the output layer. However, this causes certain drawbacks; e.g.,
careful hyperparameter tuning is required to synchronize the feedforward and
feedback training, and frequent updates of the feedback path are usually
required than that of the feedforward path. Learning of the feedforward and
feedback networks is sufficient to make TP methods capable of training, but is
having these layer-wise autoencoders a necessary condition for TP to work? We
answer this question by presenting Fixed-Weight Difference Target Propagation
(FW-DTP) that keeps the feedback weights constant during training. We confirmed
that this simple method, which naturally resolves the abovementioned problems
of TP, can still deliver informative target values to hidden layers for a given
task; indeed, FW-DTP consistently achieves higher test performance than a
baseline, the Difference Target Propagation (DTP), on four classification
datasets. We also present a novel propagation architecture that explains the
exact form of the feedback function of DTP to analyze FW-DTP.

本篇研究提出了 Fixed-Weight Difference Target Propagation（FW-DTP）算法，该算法在训练过程中保持反馈权重不变，以解决 Target Propagation（TP）算法存在的问题。结果表明，FW-DTP 算法可以为隐层提供有用的目标值，且在四个分类数据集上显著优于 Difference Target Propagation（DTP）算法。同时，文中也解释了 DTP 算法的反馈函数形式。

固定权重差分目标传播

Fixed-Weight Difference Target Propagation

Stochastic gradient descent with backpropagation is the workhorse of
artificial neural networks. It has long been recognized that backpropagation
fails to be a biologically plausible algorithm. Fundamentally, it is a
non-local procedure -- updating one neuron's synaptic weights requires
knowledge of synaptic weights or receptive fields of downstream neurons. This
limits the use of artificial neural networks as a tool for understanding the
biological principles of information processing in the brain. Lillicrap et al.
(2016) propose a more biologically plausible "feedback alignment" algorithm
that uses random and fixed backpropagation weights, and show promising
simulations. In this paper we study the mathematical properties of the feedback
alignment procedure by analyzing convergence and alignment for two-layer
networks under squared error loss. In the overparameterized setting, we prove
that the error converges to zero exponentially fast, and also that
regularization is necessary in order for the parameters to become aligned with
the random backpropagation weights. Simulations are given that are consistent
with this analysis and suggest further generalizations. These results
contribute to our understanding of how biologically plausible algorithms might
carry out weight learning in a manner different from Hebbian learning, with
performance that is comparable with the full non-local backpropagation
algorithm.

该论文研究 “反馈对齐” 算法的数学特性，通过分析二层网络在平方误差损失下的收敛和对齐，证明在过度参数化的情况下，误差会以指数速度收敛，以及参数对齐需要正则化。该成果对我们理解生物学可行的算法如何不同于 Hebbian 学习方法，具有与非本地反向传播算法相当的性能提供了方法。