Temporal difference (TD) learning algorithms with neural network function
parameterization have well-established empirical success in many practical
large-scale reinforcement learning tasks. However, theoretical understanding of
these algorithms remains challenging due to the nonlinearity of the
action-value approximation. In this paper, we develop an improved
non-asymptotic analysis of the neural TD method with a general $L$-layer neural
network. New proof techniques are developed and an improved new
$\tilde{\mathcal{O}}(\epsilon^{-1})$ sample complexity is derived. To our best
knowledge, this is the first finite-time analysis of neural TD that achieves an
$\tilde{\mathcal{O}}(\epsilon^{-1})$ complexity under the Markovian sampling,
as opposed to the best known $\tilde{\mathcal{O}}(\epsilon^{-2})$ complexity in
the existing literature.

本文基于非线性的动作价值逼近，对具有神经网络函数参数化的时序差异（TD）学习算法进行改进的有限时间分析，得到了一种改进的新的样本复杂度 Ο̃(ε^(-1))，在马尔可夫采样下取得了 Ο̃(ε^(-1)) 的复杂度，相比现有文献中已知的 Ο̃(ε^(-2)) 复杂度是第一次实现的研究。