Model-based Reinforcement Learning (MBRL) aims to make agents more
sample-efficient, adaptive, and explainable by learning an explicit model of
the environment. While the capabilities of MBRL agents have significantly
improved in recent years, how to best learn the model is still an unresolved
question. The majority of MBRL algorithms aim at training the model to make
accurate predictions about the environment and subsequently using the model to
determine the most rewarding actions. However, recent research has shown that
model predictive accuracy is often not correlated with action quality, tracing
the root cause to the \emph{objective mismatch} between accurate dynamics model
learning and policy optimization of rewards. A number of interrelated solution
categories to the objective mismatch problem have emerged as MBRL continues to
mature as a research area. In this work, we provide an in-depth survey of these
solution categories and propose a taxonomy to foster future research.

模型驱动强化学习（MBRL）通过学习环境的显式模型，使智能体更具有样本效率、自适应性和可解释性。本研究提供了对 MBRL 中解决目标不匹配问题的多个解决方案类别的深入调查，并提出了一个分类以促进未来的研究。

模型基强化学习中解决客观差异的统一视角

A Unified View on Solving Objective Mismatch in Model-Based  Reinforcement Learning

Dynamics model learning deals with the task of inferring unknown dynamics
from measurement data and predicting the future behavior of the system. A
typical approach to address this problem is to train recurrent models. However,
predictions with these models are often not physically meaningful. Further,
they suffer from deteriorated behavior over time due to accumulating errors.
Often, simulators building on first principles are available being physically
meaningful by design. However, modeling simplifications typically cause
inaccuracies in these models. Consequently, hybrid modeling is an emerging
trend that aims to combine the best of both worlds. In this paper, we propose a
new approach to hybrid modeling, where we inform the latent states of a learned
model via a black-box simulator. This allows to control the predictions via the
simulator preventing them from accumulating errors. This is especially
challenging since, in contrast to previous approaches, access to the
simulator's latent states is not available. We tackle the task by leveraging
observers, a well-known concept from control theory, inferring unknown latent
states from observations and dynamics over time. In our learning-based setting,
we jointly learn the dynamics and an observer that infers the latent states via
the simulator. Thus, the simulator constantly corrects the latent states,
compensating for modeling mismatch caused by learning. To maintain flexibility,
we train an RNN-based residuum for the latent states that cannot be informed by
the simulator.

通过黑盒模拟器控制预测，修正模型匹配错误的混合建模新方法。