We explore fixed-horizon temporal difference (TD) methods, reinforcement
learning algorithms for a new kind of value function that predicts the sum of
rewards over a $\textit{fixed}$ number of future time steps. To learn the value
function for horizon $h$, these algorithms bootstrap from the value function
for horizon $h-1$, or some shorter horizon. Because no value function
bootstraps from itself, fixed-horizon methods are immune to the stability
problems that plague other off-policy TD methods using function approximation
(also known as "the deadly triad"). Although fixed-horizon methods require the
storage of additional value functions, this gives the agent additional
predictive power, while the added complexity can be substantially reduced via
parallel updates, shared weights, and $n$-step bootstrapping. We show how to
use fixed-horizon value functions to solve reinforcement learning problems
competitively with methods such as Q-learning that learn conventional value
functions. We also prove convergence of fixed-horizon temporal difference
methods with linear and general function approximation. Taken together, our
results establish fixed-horizon TD methods as a viable new way of avoiding the
stability problems of the deadly triad.

本文提出了一种解决强化学习中含有稳定性问题的新型算法 —— 固定时间段时序差分算法，该算法通过对当前概率及其后 $h$ 步概率分别采用 $TD$ 算法进行预测，具有一定的优势。