Current methods for training recurrent neural networks are based on
backpropagation through time, which requires storing a complete history of
network states, and prohibits updating the weights `online' (after every
timestep). Real Time Recurrent Learning (RTRL) eliminates the need for history
storage and allows for online weight updates, but does so at the expense of
computational costs that are quartic in the state size. This renders RTRL
training intractable for all but the smallest networks, even ones that are made
highly sparse.
We introduce the Sparse n-step Approximation (SnAp) to the RTRL influence
matrix, which only keeps entries that are nonzero within n steps of the
recurrent core. SnAp with n=1 is no more expensive than backpropagation, and we
find that it substantially outperforms other RTRL approximations with
comparable costs such as Unbiased Online Recurrent Optimization. For highly
sparse networks, SnAp with n=2 remains tractable and can outperform
backpropagation through time in terms of learning speed when updates are done
online. SnAp becomes equivalent to RTRL when n is large.

介绍了 Sparse n-step Approximation (SnAp) 来优化 Real Time Recurrent Learning (RTRL) 的影响矩阵，在保持计算成本低的同时提高了网络的学习速度，特别是在稀疏网络时，n=2 时的 SnAp 能够优于 backpropagation in terms of learning speed when updates are done online。

一种实用的稀疏逼近算法用于实时循环学习

A Practical Sparse Approximation for Real Time Recurrent Learning

One of the central goals of Recurrent Neural Networks (RNNs) is to learn
long-term dependencies in sequential data. Nevertheless, the most popular
training method, Truncated Backpropagation through Time (TBPTT), categorically
forbids learning dependencies beyond the truncation horizon. In contrast, the
online training algorithm Real Time Recurrent Learning (RTRL) provides
untruncated gradients, with the disadvantage of impractically large
computational costs. Recently published approaches reduce these costs by
providing noisy approximations of RTRL. We present a new approximation
algorithm of RTRL, Optimal Kronecker-Sum Approximation (OK). We prove that OK
is optimal for a class of approximations of RTRL, which includes all approaches
published so far. Additionally, we show that OK has empirically negligible
noise: Unlike previous algorithms it matches TBPTT in a real world task
(character-level Penn TreeBank) and can exploit online parameter updates to
outperform TBPTT in a synthetic string memorization task. Code availiable on
github.

研究 RNN 的训练方法，提出新的 RTRL 逼近算法 OK-Kronecker-Sum 并证明其优秀性能，经实验验证 OK- Kronecker-Sum 能够在真实世界任务中匹配 TBPTT，在合成字符串记忆任务中优于 TBPTTs。