We study the problem of training a Reinforcement Learning (RL) agent that is
collaborative with humans without using any human data. Although such agents
can be obtained through self-play training, they can suffer significantly from
distributional shift when paired with unencountered partners, such as humans.
To mitigate this distributional shift, we propose Maximum Entropy
Population-based training (MEP). In MEP, agents in the population are trained
with our derived Population Entropy bonus to promote both pairwise diversity
between agents and individual diversity of agents themselves, and a common best
agent is trained by paring with agents in this diversified population via
prioritized sampling. The prioritization is dynamically adjusted based on the
training progress. We demonstrate the effectiveness of our method MEP, with
comparison to Self-Play PPO (SP), Population-Based Training (PBT), Trajectory
Diversity (TrajeDi), and Fictitious Co-Play (FCP) in the Overcooked game
environment, with partners being human proxy models and real humans. A
supplementary video showing experimental results is available at
this https URL

本文提出了一种通过最大熵种群训练（MEP）来训练 RL 系统的方法，以促进代理系统间的多样性和个体代理自身的多样性，并通过优先采样对资深代理进行训练。在 Overcooked 游戏环境中，与自我游戏 PPO（SP）等方法进行比较，我们证明了 MEP 方法的有效性。