Recent work has shown that representation learning plays a critical role in
sample-efficient reinforcement learning (RL) from pixels. Unfortunately, in
real-world scenarios, representation learning is usually fragile to
task-irrelevant distractions such as variations in background or viewpoint. To
tackle this problem, we propose a novel clustering-based approach, namely
Clustering with Bisimulation Metrics (CBM), which learns robust representations
by grouping visual observations in the latent space. Specifically, CBM
alternates between two steps: (1) grouping observations by measuring their
bisimulation distances to the learned prototypes; (2) learning a set of
prototypes according to the current cluster assignments. Computing cluster
assignments with bisimulation metrics enables CBM to capture task-relevant
information, as bisimulation metrics quantify the behavioral similarity between
observations. Moreover, CBM encourages the consistency of representations
within each group, which facilitates filtering out task-irrelevant information
and thus induces robust representations against distractions. An appealing
feature is that CBM can achieve sample-efficient representation learning even
if multiple distractions exist simultaneously.Experiments demonstrate that CBM
significantly improves the sample efficiency of popular visual RL algorithms
and achieves state-of-the-art performance on both multiple and single
distraction settings. The code is available at
this https URL.

通过聚类与双模拟度量相结合的方式，提出了一种新颖的方法 (Clustering with Bisimulation Metrics, CBM)，用于学习视觉信息的鲁棒表示，从而提高强化学习的样本效率。实验证明，CBM 显著改善了流行的视觉强化学习算法的样本效率，并在多个或单个干扰设置下实现了最先进的性能。