We propose a novel Bayesian-Optimistic Frequentist Upper Confidence Bound (BOF-UCB) algorithm for stochastic contextual linear bandits in non-stationary environments. This unique combination of Bayesian and frequentist principles enhances adaptability and performance in dynamic settings. The BOF-UCB algorithm utilizes sequential Bayesian updates to infer the posterior distribution of the unknown regression parameter, and subsequently employs a frequentist approach to compute the Upper Confidence Bound (UCB) by maximizing the expected reward over the posterior distribution. We provide theoretical guarantees of BOF-UCB's performance and demonstrate its effectiveness in balancing exploration and exploitation on synthetic datasets and classical control tasks in a reinforcement learning setting. Our results show that BOF-UCB outperforms existing methods, making it a promising solution for sequential decision-making in non-stationary environments.

我们提出了一种新颖的贝叶斯-乐观频率上置信界（BOF-UCB）算法，用于非平稳环境中的随机上下文线性臂问题。该算法通过使用顺序贝叶斯更新来推断未知回归参数的后验分布，并随后采用频率学方法通过最大化后验分布上的预期奖励来计算上置信界（UCB），从而提高了适应性和性能。我们对BOF-UCB的性能提供了理论保证，并证明了它在合成数据集和经典控制任务中在强化学习环境中平衡探索和利用方面的有效性。我们的结果表明，BOF-UCB优于现有方法，成为非平稳环境中顺序决策的有希望解决方案。

BOF-UCB：一种非平稳上下文多臂赌博机问题的贝叶斯优化频率法算法