There is a recent trend of applying multi-agent reinforcement learning (MARL)
to train an agent that can cooperate with humans in a zero-shot fashion without
using any human data. The typical workflow is to first repeatedly run self-play
(SP) to build a policy pool and then train the final adaptive policy against
this pool. A crucial limitation of this framework is that every policy in the
pool is optimized w.r.t. the environment reward function, which implicitly
assumes that the testing partners of the adaptive policy will be precisely
optimizing the same reward function as well. However, human objectives are
often substantially biased according to their own preferences, which can differ
greatly from the environment reward. We propose a more general framework,
Hidden-Utility Self-Play (HSP), which explicitly models human biases as hidden
reward functions in the self-play objective. By approximating the reward space
as linear functions, HSP adopts an effective technique to generate an augmented
policy pool with biased policies. We evaluate HSP on the Overcooked benchmark.
Empirical results show that our HSP method produces higher rewards than
baselines when cooperating with learned human models, manually scripted
policies, and real humans. The HSP policy is also rated as the most assistive
policy based on human feedback.

本研究针对多智能体强化学习在与人合作零 - shot 学习过程中的十分关键的限制，并提出了一种更通用的 Hidden-Utility Self-Play (HSP) 方法，该方法显式地模拟人类的偏好作为自我博弈的隐藏奖励函数。通过评估 Overcooked benchmark，HSP 方法成功获得了更高的协同收益，并被受试者评选为最有帮助的策略。