We propose and theoretically analyze an approach for planning with an
approximate model in reinforcement learning that can reduce the adverse impact
of model error. If the model is accurate enough, it accelerates the convergence
to the true value function too. One of its key components is the MaxEnt Model
Correction (MoCo) procedure that corrects the model's next-state distributions
based on a Maximum Entropy density estimation formulation. Based on MoCo, we
introduce the Model Correcting Value Iteration (MoCoVI) algorithm, and its
sampled-based variant MoCoDyna. We show that MoCoVI and MoCoDyna's convergence
can be much faster than the conventional model-free algorithms. Unlike
traditional model-based algorithms, MoCoVI and MoCoDyna effectively utilize an
approximate model and still converge to the correct value function.

我们提出并理论分析了一种在强化学习中使用近似模型进行规划的方法，可以减少模型误差的不利影响，并且如果模型足够准确，还可以加速收敛到真实值函数。其中一个关键组成部分是基于最大熵密度估计形式的 MaxEnt 模型校正（MoCo）过程，该过程根据下一个状态的分布来纠正模型。基于 MoCo，我们引入了模型修正值迭代（MoCoVI）算法及其基于采样的变体 MoCoDyna。我们表明，MoCoVI 和 MoCoDyna 的收敛速度比传统的无模型算法要快得多。与传统的模型算法不同，MoCoVI 和 MoCoDyna 有效地利用近似模型，并且仍然收敛到正确的值函数。

强化学习中的最大熵模型修正

Maximum Entropy Model Correction in Reinforcement Learning

Dyna-style model-based reinforcement learning contains two phases: model
rollouts to generate sample for policy learning and real environment
exploration using current policy for dynamics model learning. However, due to
the complex real-world environment, it is inevitable to learn an imperfect
dynamics model with model prediction error, which can further mislead policy
learning and result in sub-optimal solutions. In this paper, we propose
$\texttt{COPlanner}$, a planning-driven framework for model-based methods to
address the inaccurately learned dynamics model problem with conservative model
rollouts and optimistic environment exploration. $\texttt{COPlanner}$ leverages
an uncertainty-aware policy-guided model predictive control (UP-MPC) component
to plan for multi-step uncertainty estimation. This estimated uncertainty then
serves as a penalty during model rollouts and as a bonus during real
environment exploration respectively, to choose actions. Consequently,
$\texttt{COPlanner}$ can avoid model uncertain regions through conservative
model rollouts, thereby alleviating the influence of model error.
Simultaneously, it explores high-reward model uncertain regions to reduce model
error actively through optimistic real environment exploration.
$\texttt{COPlanner}$ is a plug-and-play framework that can be applied to any
dyna-style model-based methods. Experimental results on a series of
proprioceptive and visual continuous control tasks demonstrate that both sample
efficiency and asymptotic performance of strong model-based methods are
significantly improved combined with $\texttt{COPlanner}$.

提出了一种基于规划的框架 $	exttt {COPlanner}$，通过保守的模型展开和乐观的环境探索，解决了模型误差问题，提高了模型强化学习方法的样本效率和渐近性能。

COPlanner: 保守推进并乐观探索的模型引导强化学习规划器

COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically  for Model-Based RL

The idea of using machine learning (ML) methods to reconstruct the dynamics
of a system is the topic of recent studies in the geosciences, in which the key
output is a surrogate model meant to emulate the dynamical model. In order to
treat sparse and noisy observations in a rigorous way, ML can be combined to
data assimilation (DA). This yields a class of iterative methods in which, at
each iteration a DA step assimilates the observations, and alternates with a ML
step to learn the underlying dynamics of the DA analysis. In this article, we
propose to use this method to correct the error of an existent, knowledge-based
model. In practice, the resulting surrogate model is an hybrid model between
the original (knowledge-based) model and the ML model. We demonstrate
numerically the feasibility of the method using a two-layer, two-dimensional
quasi-geostrophic channel model. Model error is introduced by the means of
perturbed parameters. The DA step is performed using the strong-constraint
4D-Var algorithm, while the ML step is performed using deep learning tools. The
ML models are able to learn a substantial part of the model error and the
resulting hybrid surrogate models produce better short- to mid-range forecasts.
Furthermore, using the hybrid surrogate models for DA yields a significantly
better analysis than using the original model.

本研究将机器学习和数据同化相结合，提出了一种迭代方法来纠正现有知识型模型的误差，为地球科学建立了一个混合替代模型，该模型在短期到中期范围内有更好的预测表现

运用机器学习技术纠正数据同化和预测应用中的模型误差

Using machine learning to correct model error in data assimilation and  forecast applications

Data-driven models are subject to model errors due to limited and noisy
training data. Key to the application of such models in safety-critical domains
is the quantification of their model error. Gaussian processes provide such a
measure and uniform error bounds have been derived, which allow safe control
based on these models. However, existing error bounds require restrictive
assumptions. In this paper, we employ the Gaussian process distribution and
continuity arguments to derive a novel uniform error bound under weaker
assumptions. Furthermore, we demonstrate how this distribution can be used to
derive probabilistic Lipschitz constants and analyze the asymptotic behavior of
our bound. Finally, we derive safety conditions for the control of unknown
dynamical systems based on Gaussian process models and evaluate them in
simulations of a robotic manipulator.

本文利用高斯过程分布和连续性论证推导出一种新的弱假设下的统一误差界，分析了概率 Lipschitz 常数的演变和 Gaussian Process 模型控制未知动态系统的安全条件，且采用模拟程序验证了此方法的有效性。