This paper introduces a novel multi-armed bandits framework, termed
Contextual Restless Bandits (CRB), for complex online decision-making. This CRB
framework incorporates the core features of contextual bandits and restless
bandits, so that it can model both the internal state transitions of each arm
and the influence of external global environmental contexts. Using the dual
decomposition method, we develop a scalable index policy algorithm for solving
the CRB problem, and theoretically analyze the asymptotical optimality of this
algorithm. In the case when the arm models are unknown, we further propose a
model-based online learning algorithm based on the index policy to learn the
arm models and make decisions simultaneously. Furthermore, we apply the
proposed CRB framework and the index policy algorithm specifically to the
demand response decision-making problem in smart grids. The numerical
simulations demonstrate the performance and efficiency of our proposed CRB
approaches.

介绍了一种新的多臂赌博机框架 —— 上下文不安定赌博机（CRB），用于复杂的在线决策。该 CRB 框架结合了上下文赌博机和不安定赌博机的核心特征，可以模拟每个臂的内部状态转换以及外部全局环境上下文的影响。使用双重分解方法，我们开发了一个可扩展的指标策略算法来解决 CRB 问题，并对该算法的渐近最优性进行了理论分析。在臂模型未知的情况下，我们进一步提出了一种基于指标策略的基于模型的在线学习算法，同时学习臂模型和做出决策。此外，我们将所提出的 CRB 框架和指标策略算法专门应用于智能电网中的需求响应决策问题。数值模拟实验证明了我们所提出的 CRB 方法的性能和效率。