We consider off-policy temporal-difference (TD) learning methods for policy
evaluation in Markov decision processes with finite spaces and discounted
reward criteria, and we present a collection of convergence results for several
gradient-based TD algorithms with linear function approximation. The algorithms
we analyze include: (i) two basic forms of two-time-scale gradient-based TD
algorithms, which we call GTD and which minimize the mean squared projected
Bellman error using stochastic gradient-descent; (ii) their "robustified"
biased variants; (iii) their mirror-descent versions which combine the
mirror-descent idea with TD learning; and (iv) a single-time-scale version of
GTD that solves minimax problems formulated for approximate policy evaluation.
We derive convergence results for three types of stepsizes: constant
stepsize, slowly diminishing stepsize, as well as the standard type of
diminishing stepsize with a square-summable condition. For the first two types
of stepsizes, we apply the weak convergence method from stochastic
approximation theory to characterize the asymptotic behavior of the algorithms,
and for the standard type of stepsize, we analyze the algorithmic behavior with
respect to a stronger mode of convergence, almost sure convergence. Our
convergence results are for the aforementioned TD algorithms with three general
ways of setting their $\lambda$-parameters: (i) state-dependent $\lambda$; (ii)
a recently proposed scheme of using history-dependent $\lambda$ to keep the
eligibility traces of the algorithms bounded while allowing for relatively
large values of $\lambda$; and (iii) a composite scheme of setting the
$\lambda$-parameters that combines the preceding two schemes and allows a
broader class of generalized Bellman operators to be used for approximate
policy evaluation with TD methods.

本文考虑了有限状态和折扣回报标准下的马尔科夫决策过程策略评估问题中的离策略时间差分 (TD) 学习方法，并针对几个基于梯度的 TD 算法提出了一组收敛性结果。