This paper studies model-based reinforcement learning (RL) for regret
minimization. We focus on finite-horizon episodic RL where the transition model
$P$ belongs to a known family of models $\mathcal{P}$, a special case of which
is when models in $\mathcal{P}$ take the form of linear mixtures: $P_{\theta} =
\sum_{i=1}^{d} \theta_{i}P_{i}$. We propose a model based RL algorithm that is
based on optimism principle: In each episode, the set of models that are
`consistent' with the data collected is constructed. The criterion of
consistency is based on the total squared error of that the model incurs on the
task of predicting \emph{values} as determined by the last value estimate along
the transitions. The next value function is then chosen by solving the
optimistic planning problem with the constructed set of models. We derive a
bound on the regret, which, in the special case of linear mixtures, the regret
bound takes the form $\tilde{\mathcal{O}}(d\sqrt{H^{3}T})$, where $H$, $T$ and
$d$ are the horizon, total number of steps and dimension of $\theta$,
respectively. In particular, this regret bound is independent of the total
number of states or actions, and is close to a lower bound
$\Omega(\sqrt{HdT})$. For a general model family $\mathcal{P}$, the regret
bound is derived using the notion of the so-called Eluder dimension proposed by
Russo & Van Roy (2014).

本文研究基于模型的强化学习中的后悔最小化问题，提出一种基于乐观主义原则和线性混合模型的算法，并推导出一些后悔界的理论结果。