Temporal models based on recurrent neural networks have proven to be quite
powerful in a wide variety of applications. However, training these models
often relies on back-propagation through time, which entails unfolding the
network over many time steps, making the process of conducting credit
assignment considerably more challenging. Furthermore, the nature of
back-propagation itself does not permit the use of non-differentiable
activation functions and is inherently sequential, making parallelization of
the underlying training process difficult. Here, we propose the Parallel
Temporal Neural Coding Network (P-TNCN), a biologically inspired model trained
by the learning algorithm we call Local Representation Alignment. It aims to
resolve the difficulties and problems that plague recurrent networks trained by
back-propagation through time. The architecture requires neither unrolling in
time nor the derivatives of its internal activation functions. We compare our
model and learning procedure to other back-propagation through time
alternatives (which also tend to be computationally expensive), including
real-time recurrent learning, echo state networks, and unbiased online
recurrent optimization. We show that it outperforms these on sequence modeling
benchmarks such as Bouncing MNIST, a new benchmark we denote as Bouncing
NotMNIST, and Penn Treebank. Notably, our approach can in some instances
outperform full back-propagation through time as well as variants such as
sparse attentive back-tracking. Significantly, the hidden unit correction phase
of P-TNCN allows it to adapt to new datasets even if its synaptic weights are
held fixed (zero-shot adaptation) and facilitates retention of prior generative
knowledge when faced with a task sequence. We present results that show the
P-TNCN's ability to conduct zero-shot adaptation and online continual sequence
modeling.

本文提出了一种基于局部表示对齐算法训练的并行时间神经编码网络 (P-TNCN)，用以解决通过时间反向传播所存在的困难和问题，比如序列建模基准测试中表现优异，具有零 - shot 适应能力和在线连续序列建模的能力。

通过本地对齐分布式表示进行循环神经网络的持续学习

Continual Learning of Recurrent Neural Networks by Locally Aligning  Distributed Representations

Finding biologically plausible alternatives to back-propagation of errors is
a fundamentally important challenge in artificial neural network research. In
this paper, we propose a learning algorithm called error-driven Local
Representation Alignment (LRA-E), which has strong connections to predictive
coding, a theory that offers a mechanistic way of describing neurocomputational
machinery. In addition, we propose an improved variant of Difference Target
Propagation, another procedure that comes from the same family of algorithms as
LRA-E. We compare our procedures to several other biologically-motivated
algorithms, including two feedback alignment algorithms and Equilibrium
Propagation. In two benchmarks, we find that both of our proposed algorithms
yield stable performance and strong generalization compared to other competing
back-propagation alternatives when training deeper, highly nonlinear networks,
with LRA-E performing the best overall.

提出了一种名为误差驱动局部表示对齐 (LRA-E) 的学习算法，它与预测编码有强烈的联系，可以在训练深度非线性网络时得到更稳定的性能和更强的泛化能力，并且通过与其他生物启发算法的比较，表现最佳。