We consider the problem of quantifying uncertainty over expected cumulative rewards in model-based reinforcement learning. In particular, we focus on characterizing the variance over values induced by a distribution over MDPs. Previous work upper bounds the posterior variance over values by solving a so-called uncertainty Bellman equation, but the over-approximation may result in inefficient exploration. We propose a new uncertainty Bellman equation whose solution converges to the true posterior variance over values and explicitly characterizes the gap in previous work. Moreover, our uncertainty quantification technique is easily integrated into common exploration strategies and scales naturally beyond the tabular setting by using standard deep reinforcement learning architectures. Experiments in difficult exploration tasks, both in tabular and continuous control settings, show that our sharper uncertainty estimates improve sample-efficiency.

在模型基强化学习中，我们考虑了如何量化累积奖励的不确定性，并提出了一种新的不确定Bellman方程来弥补现有工作的不足，该方法能够更准确地告诉我们此前探索的不足。实验表明，这种更精确的不确定性估计方法能够提高样本效率。

基于模型的价值函数不确定性