Model-free Reinforcement Learning (RL) works well when experience can be
collected cheaply and model-based RL is effective when system dynamics can be
modeled accurately. However, both assumptions can be violated in real world
problems such as robotics, where querying the system can be expensive and
real-world dynamics can be difficult to model. In contrast to RL, Model
Predictive Control (MPC) algorithms use a simulator to optimize a simple policy
class online, constructing a closed-loop controller that can effectively
contend with real-world dynamics. MPC performance is usually limited by factors
such as model bias and the limited horizon of optimization. In this work, we
present a novel theoretical connection between information theoretic MPC and
entropy regularized RL and develop a Q-learning algorithm that can leverage
biased models. We validate the proposed algorithm on sim-to-sim control tasks
to demonstrate the improvements over optimal control and reinforcement learning
from scratch. Our approach paves the way for deploying reinforcement learning
algorithms on real systems in a systematic manner.

本研究提出了一种基于信息理论模型预测控制和熵正则化强化学习的 Q 学习算法，可以利用有偏模型，并在模拟控制任务中验证了该算法的有效性。

信息论模型预测 Q 学习

Information Theoretic Model Predictive Q-Learning

Many machine learning algorithms are trained and evaluated by splitting data
from a single source into training and test sets. While such focus on
in-distribution learning scenarios has led to interesting advancement, it has
not been able to tell if models are relying on dataset biases as shortcuts for
successful prediction (e.g., using snow cues for recognising snowmobiles),
resulting in biased models that fail to generalise when the bias shifts to a
different class. The cross-bias generalisation problem has been addressed by
de-biasing training data through augmentation or re-sampling, which are often
prohibitive due to the data collection cost (e.g., collecting images of a
snowmobile on a desert) and the difficulty of quantifying or expressing biases
in the first place. In this work, we propose a novel framework to train a
de-biased representation by encouraging it to be different from a set of
representations that are biased by design. This tactic is feasible in many
scenarios where it is much easier to define a set of biased representations
than to define and quantify bias. We demonstrate the efficacy of our method
across a variety of synthetic and real-world biases; our experiments show that
the method discourages models from taking bias shortcuts, resulting in improved
generalisation. Source code is available at this https URL

本文介绍了一种用于训练去偏置表示的新框架，通过鼓励其与一组从设计上存在偏差的表示不同来实现。该方法可在许多场景中使用，可以提高模型在不同类别中的泛化能力。