We consider un-discounted reinforcement learning (RL) in Markov decision processes (MDPs) under drifting non-stationarity, i.e., both the reward and state transition distributions are allowed to evolve over time, as long as their respective total variations, quantified by suitable metrics, do not exceed certain variation budgets. We first develop the Sliding Window Upper-Confidence bound for Reinforcement Learning with Confidence Widening (SWUCRL2-CW) algorithm, and establish its dynamic regret bound when the variation budgets are known. In addition, we propose the Bandit-over-Reinforcement Learning (BORL) algorithm to adaptively tune the SWUCRL2-CW algorithm to achieve the same dynamic regret bound, but in a parameter-free manner, i.e., without knowing the variation budgets. Notably, learning non-stationary MDPs via the conventional optimistic exploration technique presents a unique challenge absent in existing (non-stationary) bandit learning settings. We overcome the challenge by a novel confidence widening technique that incorporates additional optimism.

本研究使用不打折扣的强化学习方法，针对马尔可夫决策过程中的漂移非稳定性问题，提出了Sliding Window Upper-Confidence bound for Reinforcement Learning with Confidence Widening算法和Bandit-over-Reinforcement Learning算法，并证明了它们的动态后悔限制及其在参数不确定性情况下的适应性。

非静态马尔科夫决策过程的强化学习：（更多）乐观的祝福