We provide a framework for incorporating robustness -- to perturbations in the transition dynamics which we refer to as model misspecification -- into continuous control Reinforcement Learning (RL) algorithms. We specifically focus on incorporating robustness into a state-of-the-art continuous control RL algorithm called Maximum a-posteriori Policy Optimization (MPO). We achieve this by learning a policy that optimizes for a worst case, entropy-regularized, expected return objective and derive a corresponding robust entropy-regularized Bellman contraction operator. In addition, we introduce a less conservative, soft-robust, entropy-regularized objective with a corresponding Bellman operator. We show that both, robust and soft-robust policies, outperform their non-robust counterparts in nine Mujoco domains with environment perturbations. Finally, we present multiple investigative experiments that provide a deeper insight into the robustness framework; including an adaptation to another continuous control RL algorithm as well as comparing this approach to domain randomization. Performance videos can be found online at https://sites.google.com/view/robust-rl.

提供了一个框架，可将稳健性合并到持续控制强化学习算法中，通过学习最优策略并推导相应的鲁棒性熵正则化贝尔曼收缩算子来实现，并进一步引入了较不保守的软稳健熵正则化目标及相应贝尔曼算子，实验证明在九个领域的环境扰动及高维度的机器人控制方面，稳健和软稳健策略均优于无鲁棒性策略，并提供了多个探究实验来深入理解框架的其它特性。

具有模型误差的连续控制强化学习