Unsupervised reinforcement learning aims at learning a generalist policy in a
reward-free manner for fast adaptation to downstream tasks. Most of the
existing methods propose to provide an intrinsic reward based on surprise.
Maximizing or minimizing surprise drives the agent to either explore or gain
control over its environment. However, both strategies rely on a strong
assumption: the entropy of the environment's dynamics is either high or low.
This assumption may not always hold in real-world scenarios, where the entropy
of the environment's dynamics may be unknown. Hence, choosing between the two
objectives is a dilemma. We propose a novel yet simple mixture of policies to
address this concern, allowing us to optimize an objective that simultaneously
maximizes and minimizes the surprise. Concretely, we train one mixture
component whose objective is to maximize the surprise and another whose
objective is to minimize the surprise. Hence, our method does not make
assumptions about the entropy of the environment's dynamics. We call our method
a $\textbf{M}\text{ixture }\textbf{O}\text{f
}\textbf{S}\text{urprise}\textbf{S}$ (MOSS) for unsupervised reinforcement
learning. Experimental results show that our simple method achieves
state-of-the-art performance on the URLB benchmark, outperforming previous pure
surprise maximization-based objectives. Our code is available at:
this https URL.

提出了一种名为 MOSS 的策略学习方法，通过训练一个同时最大化和最小化惊喜的混合组件来解决先前基于最大惊喜或最小惊喜的方法在真实世界环境中遇到的对环境动态熵的假设问题。实验结果表明，该方法在无监督强化学习领域达到了最新的最优表现。