Model-based reinforcement learning (RL) is considered to be a promising
approach to reduce the sample complexity that hinders model-free RL. However,
the theoretical understanding of such methods has been rather limited. This
paper introduces a novel algorithmic framework for designing and analyzing
model-based RL algorithms with theoretical guarantees. We design a
meta-algorithm with a theoretical guarantee of monotone improvement to a local
maximum of the expected reward. The meta-algorithm iteratively builds a lower
bound of the expected reward based on the estimated dynamical model and sample
trajectories, and then maximizes the lower bound jointly over the policy and
the model. The framework extends the optimism-in-face-of-uncertainty principle
to non-linear dynamical models in a way that requires \textit{no explicit}
uncertainty quantification. Instantiating our framework with simplification
gives a variant of model-based RL algorithms Stochastic Lower Bounds
Optimization (SLBO). Experiments demonstrate that SLBO achieves
state-of-the-art performance when only one million or fewer samples are
permitted on a range of continuous control benchmark tasks.

本文提出了一种新的算法框架来设计和分析具有理论保证的基于模型的强化学习算法，该框架将不确定性原则扩展到非线性动态模型，其中模型 SLBO 在连续控制基准任务上实现了最先进的性能。