The Whittle index policy is a heuristic that has shown remarkably good
performance (with guaranteed asymptotic optimality) when applied to the class
of problems known as Restless Multi-Armed Bandit Problems (RMABPs). In this
paper we present QWI and QWINN, two reinforcement learning algorithms,
respectively tabular and deep, to learn the Whittle index for the total
discounted criterion. The key feature is the use of two time-scales, a faster
one to update the state-action Q -values, and a relatively slower one to update
the Whittle indices. In our main theoretical result we show that QWI, which is
a tabular implementation, converges to the real Whittle indices. We then
present QWINN, an adaptation of QWI algorithm using neural networks to compute
the Q -values on the faster time-scale, which is able to extrapolate
information from one state to another and scales naturally to large state-space
environments. For QWINN, we show that all local minima of the Bellman error are
locally stable equilibria, which is the first result of its kind for DQN-based
schemes. Numerical computations show that QWI and QWINN converge faster than
the standard Q -learning algorithm, neural-network based approximate Q-learning
and other state of the art algorithms.

使用 QWI 和 QWINN 两种强化学习算法，分别基于表格和深度学习结构，学习总折扣标准下的 Whittle 指数。关键特征是使用两个时间尺度，一个更快的时间尺度用于更新状态 - 动作 Q 值，一个相对较慢的时间尺度用于更新 Whittle 指数。在理论结果中证明了 QWI 的收敛性，并介绍了基于神经网络的 QWINN 算法，能够在大状态空间环境中从一个状态推断出另一个状态，并且与其他算法相比收敛速度更快。