Using a model of the environment and a value function, an agent can construct
many estimates of a state's value, by unrolling the model for different lengths
and bootstrapping with its value function. Our key insight is that one can
treat this set of value estimates as a type of ensemble, which we call an
\emph{implicit value ensemble} (IVE). Consequently, the discrepancy between
these estimates can be used as a proxy for the agent's epistemic uncertainty;
we term this signal \emph{model-value inconsistency} or
\emph{self-inconsistency} for short. Unlike prior work which estimates
uncertainty by training an ensemble of many models and/or value functions, this
approach requires only the single model and value function which are already
being learned in most model-based reinforcement learning algorithms. We provide
empirical evidence in both tabular and function approximation settings from
pixels that self-inconsistency is useful (i) as a signal for exploration, (ii)
for acting safely under distribution shifts, and (iii) for robustifying
value-based planning with a learned model.

本文提出了一种新的基于模型的强化学习算法，通过利用值函数，将模型不同长度的展开作为一种集合，称之为 “implicit value ensemble”，并应用于探索、在分布变化时安全操作以及学习模型规划等方面，实证结果表明这种方法对提升探索效率和增强可靠性有显著的效果。