In reinforcement learning, unsupervised skill discovery aims to learn diverse
skills without extrinsic rewards. Previous methods discover skills by
maximizing the mutual information (MI) between states and skills. However, such
an MI objective tends to learn simple and static skills and may hinder
exploration. In this paper, we propose a novel unsupervised skill discovery
method through contrastive learning among behaviors, which makes the agent
produce similar behaviors for the same skill and diverse behaviors for
different skills. Under mild assumptions, our objective maximizes the MI
between different behaviors based on the same skill, which serves as an upper
bound of the previous MI objective. Meanwhile, our method implicitly increases
the state entropy to obtain better state coverage. We evaluate our method on
challenging mazes and continuous control tasks. The results show that our
method generates diverse and far-reaching skills, and also obtains competitive
performance in downstream tasks compared to the state-of-the-art methods.

本文提出了一种基于对比学习的无监督技能发现方法，通过得到相似的行为来表征同一种技能， 并使得不同技能产生不同的行为，同时增加状态熵以获得更好的状态覆盖率，实验结果表明，该方法能够产生各种远程技能，并在下游任务中达到有竞争力的表现。

行为对比学习：无监督技能发现

Behavior Contrastive Learning for Unsupervised Skill Discovery

Recent exploration methods have proven to be a recipe for improving
sample-efficiency in deep reinforcement learning (RL). However, efficient
exploration in high-dimensional observation spaces still remains a challenge.
This paper presents Random Encoders for Efficient Exploration (RE3), an
exploration method that utilizes state entropy as an intrinsic reward. In order
to estimate state entropy in environments with high-dimensional observations,
we utilize a k-nearest neighbor entropy estimator in the low-dimensional
representation space of a convolutional encoder. In particular, we find that
the state entropy can be estimated in a stable and compute-efficient manner by
utilizing a randomly initialized encoder, which is fixed throughout training.
Our experiments show that RE3 significantly improves the sample-efficiency of
both model-free and model-based RL methods on locomotion and navigation tasks
from DeepMind Control Suite and MiniGrid benchmarks. We also show that RE3
allows learning diverse behaviors without extrinsic rewards, effectively
improving sample-efficiency in downstream tasks. Source code and videos are
available at this https URL

本文介绍了一种基于状态熵作为内在奖励的随机编码器（Random Encoders for Efficient Exploration，RE3）的探索方法，旨在解决高维观察空间中的有效探索问题，并在 DeepMind Control Suite 和 MiniGrid 基准测试的行动和导航任务中得到了显著的样本效率改进，并且在无外部奖励的情况下，可以学习多样化的行为。