Truncated backpropagation through time (TBPTT) is a popular method for learning in recurrent neural networks (RNNs) that saves computation and memory at the cost of bias by truncating backpropagation after a fixed number of lags. In practice, choosing the optimal truncation length is difficult: TBPTT will not converge if the truncation length is too small, or will converge slowly if it is too large. We propose an adaptive TBPTT scheme that converts the problem from choosing a temporal lag to one of choosing a tolerable amount of gradient bias. For many realistic RNNs, the TBPTT gradients decay geometrically for large lags; under this condition, we can control the bias by varying the truncation length adaptively. For RNNs with smooth activation functions, we prove that this bias controls the convergence rate of SGD with biased gradients for our non-convex loss. Using this theory, we develop a practical method for adaptively estimating the truncation length during training. We evaluate our adaptive TBPTT method on synthetic data and language modeling tasks and find that our adaptive TBPTT ameliorates the computational pitfalls of fixed TBPTT.

本文提出了一种适应性的TBPTT方案，将时间滞后问题转化为选择一定容忍梯度偏差量的问题，并对于满足几何衰减的真实RNNs，可以通过变化截断长度来控制偏差，证明了该偏差控制了含非凸损失函数的SGD的收敛速度，并设计了一种实用的适应性截断长度的方法。在语言模型任务上的实验表明，适应性TBPTT减轻了固定TBPTT的计算缺陷。

自适应截断反向传播通过时间来控制梯度偏差