Temporal-Difference learning (TD) [Sutton, 1988] with function approximation
can converge to solutions that are worse than those obtained by Monte-Carlo
regression, even in the simple case of on-policy evaluation. To increase our
understanding of the problem, we investigate the issue of approximation errors
in areas of sharp discontinuities of the value function being further
propagated by bootstrap updates. We show empirical evidence of this leakage
propagation, and show analytically that it must occur, in a simple Markov
chain, when function approximation errors are present. For reversible policies,
the result can be interpreted as the tension between two terms of the loss
function that TD minimises, as recently described by [Ollivier, 2018]. We show
that the upper bounds from [Tsitsiklis and Van Roy, 1997] hold, but they do not
imply that leakage propagation occurs and under what conditions. Finally, we
test whether the problem could be mitigated with a better state representation,
and whether it can be learned in an unsupervised manner, without rewards or
privileged information.

本研究探讨用函数逼近的时序差分学习论（TD）可收敛至比蒙特卡罗回归更劣的解的问题，以及针对价值函数在出现急剧不连续的地方的逼近误差在自举更新中何以进一步扩散的问题。我们通过实证找到了泄漏扩散的证据，并论证了仅当逼近误差时，这种情况会出现。最后，我们证明了泄漏传播从 [Tsitsiklis and Van Roy, 1997] 中得出，但是这并不意味着泄漏传播会发生以及何种情况下会发生，最后，我们测试了这个问题是否可以通过更好的状态表示来缓解，并且是否可以在无奖励或特权信息的情况下进行学习。