Restless multi-armed bandits (RMABs) are used to model sequential resource
allocation in public health intervention programs. In these settings, the
underlying transition dynamics are often unknown a priori, requiring online
reinforcement learning (RL). However, existing methods in online RL for RMABs
cannot incorporate properties often present in real-world public health
applications, such as contextual information and non-stationarity. We present
Bayesian Learning for Contextual RMABs (BCoR), an online RL approach for RMABs
that novelly combines techniques in Bayesian modeling with Thompson sampling to
flexibly model a wide range of complex RMAB settings, such as contextual and
non-stationary RMABs. A key contribution of our approach is its ability to
leverage shared information within and between arms to learn unknown RMAB
transition dynamics quickly in budget-constrained settings with relatively
short time horizons. Empirically, we show that BCoR achieves substantially
higher finite-sample performance than existing approaches over a range of
experimental settings, including one constructed from a real-world public
health campaign in India.

基于贝叶斯学习和汤普森抽样的上下文多臂赌博机在线强化学习方法可以高效建模复杂的上下文相关和非固定的公共卫生干预项目中的资源分配，具有较高的性能表现。