Discovering all useful solutions for a given task is crucial for transferable
RL agents, to account for changes in the task or transition dynamics. This is
not considered by classical RL algorithms that are only concerned with finding
the optimal policy, given the current task and dynamics. We propose a simple
method for discovering all possible solutions of a given task, to obtain an
agent that performs well in the transfer setting and adapts quickly to changes
in the task or transition dynamics. Our method iteratively learns a set of
policies, while each subsequent policy is constrained to yield a solution that
is unlikely under all previous policies. Unlike prior methods, our approach
does not require learning additional models for novelty detection and avoids
balancing task and novelty reward signals, by directly incorporating the
constraint into the action selection and optimization steps.

基于转移学习的强化学习智能体在发现给定任务的所有有用解决方案方面至关重要，以应对任务或转移动力学的变化。我们提出了一种简单的方法来发现给定任务的所有可能解决方案，以获得在转移设置中表现良好并能够快速适应任务或转移动力学变化的智能体。我们的方法迭代地学习一组策略，而每个后续策略都受到所有先前策略下的不太可能解决方案的约束。与以往的方法不同，我们的方法不需要学习额外的新颖性检测模型，并通过直接将约束融入行动选择和优化步骤中，避免了任务和新颖性奖励信号的平衡。