We introduce Contrastive Intrinsic Control (CIC), an algorithm for unsupervised skill discovery that maximizes the mutual information between skills and state transitions. In contrast to most prior approaches, CIC uses a decomposition of the mutual information that explicitly incentivizes diverse behaviors by maximizing state entropy. We derive a novel lower bound estimate for the mutual information which combines a particle estimator for state entropy to generate diverse behaviors and contrastive learning to distill these behaviors into distinct skills. We evaluate our algorithm on the Unsupervised Reinforcement Learning Benchmark, which consists of a long reward-free pre-training phase followed by a short adaptation phase to downstream tasks with extrinsic rewards. We find that CIC substantially improves over prior unsupervised skill discovery methods and outperforms the next leading overall exploration algorithm in terms of downstream task performance.

本文提出了一种无监督技能发现算法 Contrastive Intrinsic Control，它可以最大化状态转换和潜在技能向量之间的互信息。该算法对状态转换和技能之间进行对比学习，以学习行为嵌入并将这些嵌入的熵最大化作为内在奖励以促进行为多样性。我们在无监督强化学习基准上评估了我们的算法，CIC大大提高了适应效率，在无监督技能发现方法方面优于以前的方法1.79倍，并且整体探索算法效果比第二名提高1.18倍。

无监督技能发现的对比内在控制