Balancing exploration and exploitation is crucial in reinforcement learning
(RL). In this paper, we study model-based posterior sampling for reinforcement
learning (PSRL) in continuous state-action spaces theoretically and
empirically. First, we show the first regret bound of PSRL in continuous spaces
which is polynomial in the episode length to the best of our knowledge. With
the assumption that reward and transition functions can be modeled by Bayesian
linear regression, we develop a regret bound of $\tilde{O}(H^{3/2}d\sqrt{T})$,
where $H$ is the episode length, $d$ is the dimension of the state-action
space, and $T$ indicates the total time steps. This result matches the
best-known regret bound of non-PSRL methods in linear MDPs. Our bound can be
extended to nonlinear cases as well with feature embedding: using linear
kernels on the feature representation $\phi$, the regret bound becomes
$\tilde{O}(H^{3/2}d_{\phi}\sqrt{T})$, where $d_\phi$ is the dimension of the
representation space. Moreover, we present MPC-PSRL, a model-based posterior
sampling algorithm with model predictive control for action selection. To
capture the uncertainty in models, we use Bayesian linear regression on the
penultimate layer (the feature representation layer $\phi$) of neural networks.
Empirical results show that our algorithm achieves the state-of-the-art sample
efficiency in benchmark continuous control tasks compared to prior model-based
algorithms, and matches the asymptotic performance of model-free algorithms.

本文研究了连续状态动作空间中强化学习的基于模型的后验抽样（PSRL），提出了第一个后验抽样的遗憾上界，并开发了 MPC–PSRL 算法来选择动作，通过贝叶斯线性回归捕获模型中的不确定性，在基准连续控制任务中实现了最先进的样本效率，并与无模型算法的渐近性能相匹配。