In lung radiotherapy, infrared cameras can record the location of reflective
objects on the chest to infer the position of the tumor moving due to
breathing, but treatment system latencies hinder radiation beam precision.
Real-time recurrent learning (RTRL), is a potential solution as it can learn
patterns within non-stationary respiratory data but has high complexity. This
study assesses the capabilities of resource-efficient online RNN algorithms,
namely unbiased online recurrent optimization (UORO), sparse-1 step
approximation (SnAp-1), and decoupled neural interfaces (DNI) to forecast
respiratory motion during radiotherapy treatment accurately. We use time series
containing the 3D position of external markers on the chest of healthy
subjects. We propose efficient implementations for SnAp-1 and DNI based on
compression of the influence and immediate Jacobian matrices and an accurate
update of the linear coefficients used in credit assignment estimation,
respectively. The original sampling frequency was 10Hz; we performed resampling
at 3.33Hz and 30Hz. We use UORO, SnAp-1, and DNI to forecast each marker's 3D
position with horizons (the time interval in advance for which the prediction
is made) h<=2.1s and compare them with RTRL, least mean squares, and linear
regression. RNNs trained online achieved similar or better accuracy than most
previous works using larger training databases and deep learning, even though
we used only the first minute of each sequence to predict motion within that
exact sequence. SnAp-1 had the lowest normalized root mean square errors
(nRMSE) averaged over the horizon values considered, equal to 0.335 and 0.157,
at 3.33Hz and 10.0Hz, respectively. Similarly, UORO had the highest accuracy at
30Hz, with an nRMSE of 0.0897. DNI's inference time, equal to 6.8ms per time
step at 30Hz (Intel Core i7-13700 CPU), was the lowest among the RNN methods
examined.

在肺放射疗法中，红外相机可以记录胸部上反射物体的位置，以推断因呼吸而移动的肿瘤的位置，但治疗系统延迟影响了辐射光束的精确性。本研究评估了资源高效的在线 RNN 算法（无偏在线重复优化，SnAp-1 和 DNI）准确预测放射疗法治疗期间的呼吸运动的能力。

利用在线学习的循环神经网络进行呼吸运动预测，以增强外部引导放射治疗的安全性

Respiratory motion forecasting with online learning of recurrent neural  networks for safety enhancement in externally guided radiotherapy

Real-time recurrent learning (RTRL) for sequence-processing recurrent neural
networks (RNNs) offers certain conceptual advantages over backpropagation
through time (BPTT). RTRL requires neither caching past activations nor
truncating context, and enables online learning. However, RTRL's time and space
complexity make it impractical. To overcome this problem, most recent work on
RTRL focuses on approximation theories, while experiments are often limited to
diagnostic settings. Here we explore the practical promise of RTRL in more
realistic settings. We study actor-critic methods that combine RTRL and policy
gradients, and test them in several subsets of DMLab-30, ProcGen, and
Atari-2600 environments. On DMLab memory tasks, our system trained on fewer
than 1.2 B environmental frames is competitive with or outperforms well-known
IMPALA and R2D2 baselines trained on 10 B frames. To scale to such challenging
tasks, we focus on certain well-known neural architectures with element-wise
recurrence, allowing for tractable RTRL without approximation. We also discuss
rarely addressed limitations of RTRL in real-world applications, such as its
complexity in the multi-layer case.

本文研究了将实时递归学习和策略梯度相结合的演员 - 评论员方法在 DMLab、ProcGen 和 Atari-2600 环境中的应用，结果表明，在 DMLab 记忆任务中，我们的系统相比于训练了 10 B 帧的 IMPALA 和 R2D2 基线的系统，只需训练不到 1.2 B 个环境帧就能够达到很好的性能表现。

探索实时循环学习的优缺点

Exploring the Promise and Limits of Real-Time Recurrent Learning

Backpropagation through time (BPTT) is the standard algorithm for training
recurrent neural networks (RNNs), which requires separate simulation phases for
the forward and backward passes for inference and learning, respectively.
Moreover, BPTT requires storing the complete history of network states between
phases, with memory consumption growing proportional to the input sequence
length. This makes BPTT unsuited for online learning and presents a challenge
for implementation on low-resource real-time systems. Real-Time Recurrent
Learning (RTRL) allows online learning, and the growth of required memory is
independent of sequence length. However, RTRL suffers from exceptionally high
computational costs that grow proportional to the fourth power of the state
size, making RTRL computationally intractable for all but the smallest of
networks. In this work, we show that recurrent networks exhibiting high
activity sparsity can reduce the computational cost of RTRL. Moreover,
combining activity and parameter sparsity can lead to significant enough
savings in computational and memory costs to make RTRL practical. Unlike
previous work, this improvement in the efficiency of RTRL can be achieved
without using any approximations for the learning process.

本文提出，对于循环网络，高效的实时递归学习可以通过结合活动稀疏性和参数稀疏性得到显着的计算和存储成本节省，无需使用任何近似来学习过程。

通过活性和参数的稀疏组合实现高效实时递归学习

Efficient Real Time Recurrent Learning through combined activity and parameter sparsity

State construction from sensory observations is an important component of a
reinforcement learning agent. One solution for state construction is to use
recurrent neural networks. Two popular gradient-based methods for recurrent
learning are back-propagation through time (BPTT), and real-time recurrent
learning (RTRL). BPTT looks at the complete sequence of observations before
computing gradients and is unsuitable for online real-time updates. RTRL can do
online updates but scales poorly to large networks. In this paper, we propose
two constraints that make RTRL scalable. We show that by either decomposing the
network into independent modules or learning a recurrent network incrementally,
we can make RTRL scale linearly with the number of parameters. Unlike prior
scalable gradient estimation algorithms, such as UORO and Truncated-BPTT, our
algorithms do not add noise or bias to the gradient estimate. Instead, they
trade off the functional capacity of the recurrent network to achieve scalable
learning. We demonstrate the effectiveness of our approach over Truncated-BPTT
on a benchmark inspired by animal learning and in policy evaluation for expert
Rainbow-DQN agents in the Arcade Learning Environment (ALE).

这篇论文介绍了一种基于循环神经网络的状态构建方法，提出了能够让实时递归学习可扩展的两个约束条件，并在基准测试和政策评估中证明了其有效性。