We study the convergence of $\mathtt{Expected~Sarsa}(\lambda)$ with linear
function approximation. We show that applying the off-line estimate (multi-step
bootstrapping) to $\mathtt{Expected~Sarsa}(\lambda)$ is unstable for off-policy
learning. Furthermore, based on convex-concave saddle-point framework, we
propose a convergent $\mathtt{Gradient~Expected~Sarsa}(\lambda)$
($\mathtt{GES}(\lambda)$) algorithm. The theoretical analysis shows that our
$\mathtt{GES}(\lambda)$ converges to the optimal solution at a linear
convergence rate, which is comparable to extensive existing state-of-the-art
gradient temporal difference learning algorithms. Furthermore, we develop a
Lyapunov function technique to investigate how the step-size influences
finite-time performance of $\mathtt{GES}(\lambda)$, such technique of Lyapunov
function can be potentially generalized to other GTD algorithms. Finally, we
conduct experiments to verify the effectiveness of our $\mathtt{GES}(\lambda)$.

本研究针对线性函数近似下的 Expected Sarsa 算法的收敛问题，通过提出收敛性较好的 Gradient Expected Sarsa 算法，并应用 Lyapunov 函数技术分析其性能，得到较优实验结果。