Temporal difference (TD) learning is one of the main foundations of modern
reinforcement learning. This paper studies the use of TD(0), a canonical TD
algorithm, to estimate the value function of a given policy from a batch of
data. In this batch setting, we show that TD(0) may converge to an inaccurate
value function because the update following an action is weighted according to
the number of times that action occurred in the batch -- not the true
probability of the action under the given policy. To address this limitation,
we introduce \textit{policy sampling error corrected}-TD(0) (PSEC-TD(0)).
PSEC-TD(0) first estimates the empirical distribution of actions in each state
in the batch and then uses importance sampling to correct for the mismatch
between the empirical weighting and the correct weighting for updates following
each action. We refine the concept of a certainty-equivalence estimate and
argue that PSEC-TD(0) is a more data efficient estimator than TD(0) for a fixed
batch of data. Finally, we conduct an empirical evaluation of PSEC-TD(0) on
three batch value function learning tasks, with a hyperparameter sensitivity
analysis, and show that PSEC-TD(0) produces value function estimates with lower
mean squared error than TD(0).

本文针对 TD (0) 算法在批量设置下的不足之处，提出一种基于重要性采样和估算经验分布的改进算法 ——PSEC-TD (0)，并在三个价值函数学习任务上进行实证评估，结果显示 PSEC-TD (0) 比 TD (0) 具有更低的均方误差，是一种更高效的估计算法。