The stochastic multi-armed bandit problem studies decision-making under uncertainty. In the problem, the learner interacts with an environment by choosing an action at each round, where a round is an instance of an interaction. In response, the environment reveals a reward, which is sampled from a stochastic process, to the learner. The goal of the learner is to maximize cumulative reward. A specific variation of the stochastic multi-armed bandit problem is the restless bandit, where the reward for each action is sampled from a Markov chain. The restless bandit with a discrete state-space is a well-studied problem, but to the best of our knowledge, not many results exist for the continuous state-space version which has many applications such as hyperparameter optimization. In this work, we tackle the restless bandit with continuous state-space by assuming the rewards are the inner product of an action vector and a state vector generated by a linear Gaussian dynamical system. To predict the reward for each action, we propose a method that takes a linear combination of previously observed rewards for predicting each action's next reward. We show that, regardless of the sequence of previous actions chosen, the reward sampled for any previously chosen action can be used for predicting another action's future reward, i.e. the reward sampled for action 1 at round $t-1$ can be used for predicting the reward for action $2$ at round $t$. This is accomplished by designing a modified Kalman filter with a matrix representation that can be learned for reward prediction. Numerical evaluations are carried out on a set of linear Gaussian dynamical systems.

我们研究了连续状态空间中的不安宁赌博机问题，采用线性高斯动态系统生成的动作向量和状态向量的内积作为奖励，通过一种方法对每个动作的奖励进行预测，该方法通过线性组合先前观察到的奖励来预测每个动作的未来奖励。我们展示了无论先前选择的动作序列如何，可以利用为任何先前选择的动作采样的奖励来预测另一个动作的未来奖励，即$t-1$回合选择的动作1的奖励可以用于预测$t$回合的动作2的奖励。为此，我们设计了一种修改的卡尔曼滤波器，并提供了在一组线性高斯动态系统上的数值评估。

线性高斯动态系统产Generated生的不相容Bandit问题