Top-performing Model-Based Reinforcement Learning (MBRL) agents, such as
Dreamer, learn the world model by reconstructing the image observations. Hence,
they often fail to discard task-irrelevant details and struggle to handle
visual distractions. To address this issue, previous work has proposed to
contrastively learn the world model, but the performance tends to be inferior
in the absence of distractions. In this paper, we seek to enhance robustness to
distractions for MBRL agents. Specifically, we consider incorporating
prototypical representations, which have yielded more accurate and robust
results than contrastive approaches in computer vision. However, it remains
elusive how prototypical representations can benefit temporal dynamics learning
in MBRL, since they treat each image independently without capturing temporal
structures. To this end, we propose to learn the prototypes from the recurrent
states of the world model, thereby distilling temporal structures from past
observations and actions into the prototypes. The resulting model, DreamerPro,
successfully combines Dreamer with prototypes, making large performance gains
on the DeepMind Control suite both in the standard setting and when there are
complex background distractions. Code available at
this https URL .

本研究旨在提高基于模型的强化学习代理在处理视觉干扰方面的鲁棒性。我们提出了一种从世界模型的循环状态中学习原型表示的方法，将过去的观察和动作中的时间结构提取为原型。DreamerPro 模型通过将 Dreamer 与原型组合起来，在标准设置和存在复杂背景干扰的情况下都取得了大幅性能提高。