Training recurrent neural networks typically relies on backpropagation
through time (BPTT). BPTT depends on forward and backward passes to be
completed, rendering the network locked to these computations before loss
gradients are available. Recently, Jaderberg et al. proposed synthetic
gradients to alleviate the need for full BPTT. In their implementation
synthetic gradients are learned through a mixture of backpropagated gradients
and bootstrapped synthetic gradients, analogous to the temporal difference (TD)
algorithm in Reinforcement Learning (RL). However, as in TD learning, heavy use
of bootstrapping can result in bias which leads to poor synthetic gradient
estimates. Inspired by the accumulate $\mathrm{TD}(\lambda)$ in RL, we propose
a fully online method for learning synthetic gradients which avoids the use of
BPTT altogether: accumulate $BP(\lambda)$. As in accumulate
$\mathrm{TD}(\lambda)$, we show analytically that accumulate
$\mathrm{BP}(\lambda)$ can control the level of bias by using a mixture of
temporal difference errors and recursively defined eligibility traces. We next
demonstrate empirically that our model outperforms the original implementation
for learning synthetic gradients in a variety of tasks, and is particularly
suited for capturing longer timescales. Finally, building on recent work we
reflect on accumulate $\mathrm{BP}(\lambda)$ as a principle for learning in
biological circuits. In summary, inspired by RL principles we introduce an
algorithm capable of bias-free online learning via synthetic gradients.

通过累积 BP (lambda) 算法，我们提出了一个受强化学习原理启发的能实现无偏置在线学习的合成梯度算法。

BP (λ): 在线学习通过合成梯度

BP(λ): Online Learning via Synthetic Gradients

When training neural networks, the use of Synthetic Gradients (SG) allows
layers or modules to be trained without update locking - without waiting for a
true error gradient to be backpropagated - resulting in Decoupled Neural
Interfaces (DNIs). This unlocked ability of being able to update parts of a
neural network asynchronously and with only local information was demonstrated
to work empirically in Jaderberg et al (2016). However, there has been very
little demonstration of what changes DNIs and SGs impose from a functional,
representational, and learning dynamics point of view. In this paper, we study
DNIs through the use of synthetic gradients on feed-forward networks to better
understand their behaviour and elucidate their effect on optimisation. We show
that the incorporation of SGs does not affect the representational strength of
the learning system for a neural network, and prove the convergence of the
learning system for linear and deep linear models. On practical problems we
investigate the mechanism by which synthetic gradient estimators approximate
the true loss, and, surprisingly, how that leads to drastically different
layer-wise representations. Finally, we also expose the relationship of using
synthetic gradients to other error approximation techniques and find a unifying
language for discussion and comparison.

本文通过在前馈网络中使用合成梯度来研究解耦神经接口 (DNIs)，证明了合成梯度对于神经网络的表示强度没有影响，并证明了它在线性和深度线性模型中的学习系统的收敛性，并揭示了使用合成梯度估计器来近似真实损失的机制，及其如何导致截然不同的分层表示。最后，还揭示了使用合成梯度与其他误差近似技术的关系，为相关讨论和比较提供了一种统一的语言。

理解合成梯度和解耦神经接口

Understanding Synthetic Gradients and Decoupled Neural Interfaces

Training directed neural networks typically requires forward-propagating data
through a computation graph, followed by backpropagating error signal, to
produce weight updates. All layers, or more generally, modules, of the network
are therefore locked, in the sense that they must wait for the remainder of the
network to execute forwards and propagate error backwards before they can be
updated. In this work we break this constraint by decoupling modules by
introducing a model of the future computation of the network graph. These
models predict what the result of the modelled subgraph will produce using only
local information. In particular we focus on modelling error gradients: by
using the modelled synthetic gradient in place of true backpropagated error
gradients we decouple subgraphs, and can update them independently and
asynchronously i.e. we realise decoupled neural interfaces. We show results for
feed-forward models, where every layer is trained asynchronously, recurrent
neural networks (RNNs) where predicting one's future gradient extends the time
over which the RNN can effectively model, and also a hierarchical RNN system
with ticking at different timescales. Finally, we demonstrate that in addition
to predicting gradients, the same framework can be used to predict inputs,
resulting in models which are decoupled in both the forward and backwards pass
-- amounting to independent networks which co-learn such that they can be
composed into a single functioning corporation.

通过引入对神经网络图计算的未来模型，预测梯度和输入；实现神经网络各部分的解耦，异步更新；并对前馈模型、循环神经网络、分层 RNN 等进行了实验和验证。