We study the Whittle index learning algorithm for restless multi-armed bandits. We consider index learning algorithm with Q-learning. We first present Q-learning algorithm with exploration policies -- epsilon-greedy, softmax, epsilon-softmax with constant stepsizes. We extend the study of Q-learning to index learning for single-armed restless bandit. The algorithm of index learning is two-timescale variant of stochastic approximation, on slower timescale we update index learning scheme and on faster timescale we update Q-learning assuming fixed index value. In Q-learning updates are in asynchronous manner. We study constant stepsizes two timescale stochastic approximation algorithm. We provide analysis of two-timescale stochastic approximation for index learning with constant stepsizes. Further, we present study on index learning with deep Q-network (DQN) learning and linear function approximation with state-aggregation method. We describe the performance of our algorithms using numerical examples. We have shown that index learning with Q learning, DQN and function approximations learns the Whittle index.

本文研究了用于烦躁多臂老虎机的惠特尔指数学习算法，填补了现有研究在该领域的空白。作者提出了一种结合Q学习与探索策略的算法，并分析了其在恒定步长下的性能。实验证明，该算法可以有效学习惠特尔指数，具有广泛的应用潜力。

具有恒定步长的烦躁多臂老虎机的惠特尔指数学习算法