A key component of model-based reinforcement learning (RL) is a dynamics
model that predicts the outcomes of actions. Errors in this predictive model
can degrade the performance of model-based controllers, and complex Markov
decision processes (MDPs) can present exceptionally difficult prediction
problems. To mitigate this issue, we propose predictable MDP abstraction (PMA):
instead of training a predictive model on the original MDP, we train a model on
a transformed MDP with a learned action space that only permits predictable,
easy-to-model actions, while covering the original state-action space as much
as possible. As a result, model learning becomes easier and more accurate,
which allows robust, stable model-based planning or model-based RL. This
transformation is learned in an unsupervised manner, before any task is
specified by the user. Downstream tasks can then be solved with model-based
control in a zero-shot fashion, without additional environment interactions. We
theoretically analyze PMA and empirically demonstrate that PMA leads to
significant improvements over prior unsupervised model-based RL approaches in a
range of benchmark environments. Our code and videos are available at
this https URL

本研究提出了一种可预判马尔可夫决策过程抽象的方法，通过无监督学习在训练模型时进行转换，使得模型的学习变得更加容易和精确，从而实现强大的基于模型的规划或基于模型的强化学习，已在多项基准环境中证明了其较先前的无监督模型训练方法具有显著的改进。