In this paper, we revisit the recurrent back-propagation (RBP) algorithm,
discuss the conditions under which it applies as well as how to satisfy them in
deep neural networks. We show that RBP can be unstable and propose two variants
based on conjugate gradient on the normal equations (CG-RBP) and Neumann series
(Neumann-RBP). We further investigate the relationship between Neumann-RBP and
back propagation through time (BPTT) and its truncated version (TBPTT). Our
Neumann-RBP has the same time complexity as TBPTT but only requires constant
memory, whereas TBPTT's memory cost scales linearly with the number of
truncation steps. We examine all RBP variants along with BPTT and TBPTT in
three different application domains: associative memory with continuous
Hopfield networks, document classification in citation networks using graph
neural networks and hyperparameter optimization for fully connected networks.
All experiments demonstrate that RBPs, especially the Neumann-RBP variant, are
efficient and effective for optimizing convergent recurrent neural networks.
Code is released at: https://github.com/lrjconan/RBP.

本文重新审视了递归反向传播算法（RBP）及其适用条件，提出了基于共轭梯度和 Neumann 级数的两个改进算法，并在三个不同的应用领域进行了实验。实验证明，尤其是 Neumann-RBP 变体，对于优化收敛的循环神经网络效率高且有效。