Visual Model-Based Reinforcement Learning (MBRL) promises to encapsulate
agent's knowledge about the underlying dynamics of the environment, enabling
learning a world model as a useful planner. However, top MBRL agents such as
Dreamer often struggle with visual pixel-based inputs in the presence of
exogenous or irrelevant noise in the observation space, due to failure to
capture task-specific features while filtering out irrelevant spatio-temporal
details. To tackle this problem, we apply a spatio-temporal masking strategy, a
bisimulation principle, combined with latent reconstruction, to capture
endogenous task-specific aspects of the environment for world models,
effectively eliminating non-essential information. Joint training of
representations, dynamics, and policy often leads to instabilities. To further
address this issue, we develop a Hybrid Recurrent State-Space Model (HRSSM)
structure, enhancing state representation robustness for effective policy
learning. Our empirical evaluation demonstrates significant performance
improvements over existing methods in a range of visually complex control tasks
such as Maniskill \cite{gu2023maniskill2} with exogenous distractors from the
Matterport environment. Our code is avaliable at
this https URL

利用视觉模型仿真提升强化学习效果，通过空时屏蔽策略和混合循环状态空间模型来准确捕捉环境的重要特征，以提高策略的学习与表现。

学习潜在动态健壮表示的世界模型

Learning Latent Dynamic Robust Representations for World Models

Visual model-based reinforcement learning (RL) has the potential to enable
sample-efficient robot learning from visual observations. Yet the current
approaches typically train a single model end-to-end for learning both visual
representations and dynamics, making it difficult to accurately model the
interaction between robots and small objects. In this work, we introduce a
visual model-based RL framework that decouples visual representation learning
and dynamics learning. Specifically, we train an autoencoder with convolutional
layers and vision transformers (ViT) to reconstruct pixels given masked
convolutional features, and learn a latent dynamics model that operates on the
representations from the autoencoder. Moreover, to encode task-relevant
information, we introduce an auxiliary reward prediction objective for the
autoencoder. We continually update both autoencoder and dynamics model using
online samples collected from environment interaction. We demonstrate that our
decoupling approach achieves state-of-the-art performance on a variety of
visual robotic tasks from Meta-world and RLBench, e.g., we achieve 81.7%
success rate on 50 visual robotic manipulation tasks from Meta-world, while the
baseline achieves 67.9%. Code is available on the project website:
this https URL.

通过自编码器和动态学习的分离以及辅助奖励预测目标的引入，提出了一种新的视觉模型 RL 框架，取得了 Meta-world 和 RLBench 的 50 项任务中，81.7% 的成功率的表现。

面具世界模型的视觉控制

Masked World Models for Visual Control

The ability to plan into the future while utilizing only raw high-dimensional
observations, such as images, can provide autonomous agents with broad
capabilities. Visual model-based reinforcement learning (RL) methods that plan
future actions directly have shown impressive results on tasks that require
only short-horizon reasoning, however, these methods struggle on temporally
extended tasks. We argue that it is easier to solve long-horizon tasks by
planning sequences of states rather than just actions, as the effects of
actions greatly compound over time and are harder to optimize. To achieve this,
we draw on the idea of collocation, which has shown good results on
long-horizon tasks in optimal control literature, and adapt it to the
image-based setting by utilizing learned latent state space models. The
resulting latent collocation method (LatCo) optimizes trajectories of latent
states, which improves over previously proposed shooting methods for visual
model-based RL on tasks with sparse rewards and long-term goals. Videos and
code at this https URL

该研究使用学习的潜在状态空间模型，结合目测模型强化学习方法，提出了一种优化潜在状态轨迹的 LatCo 方法，来解决长时程、奖励稀疏的任务。通过序列规划，相比于之前使用的射击方法，该方法在历经时间过程中的效果更好。