We devise a distributional variant of gradient temporal-difference (TD)
learning. Distributional reinforcement learning has been demonstrated to
outperform the regular one in the recent study
\citep{bellemare2017distributional}. In the policy evaluation setting, we
design two new algorithms called distributional GTD2 and distributional TDC
using the Cram{\'e}r distance on the distributional version of the Bellman
error objective function, which inherits advantages of both the nonlinear
gradient TD algorithms and the distributional RL approach. In the control
setting, we propose the distributional Greedy-GQ using the similar derivation.
We prove the asymptotic almost-sure convergence of distributional GTD2 and TDC
to a local optimal solution for general smooth function approximators, which
includes neural networks that have been widely used in recent study to solve
the real-life RL problems. In each step, the computational complexities of
above three algorithms are linear w.r.t.\ the number of the parameters of the
function approximator, thus can be implemented efficiently for neural networks.

提出了分布式渐进时间差分（TD）学习的变体，并设计了新的分布式 GTD2 和分布式 TDC 算法，以及分布式 Greedy-GQ 控制设置算法。证明了分布式 GTD2 和 TDC 算法在一般光滑函数逼近器中的渐近几乎确定性收敛性。