One of the fundamental quests of AI is to produce agents that coordinate well
with humans. This problem is challenging, especially in domains that lack high
quality human behavioral data, because multi-agent reinforcement learning (RL)
often converges to different equilibria from the ones that humans prefer. We
propose a novel framework, instructRL, that enables humans to specify what kind
of strategies they expect from their AI partners through natural language
instructions. We use pretrained large language models to generate a prior
policy conditioned on the human instruction and use the prior to regularize the
RL objective. This leads to the RL agent converging to equilibria that are
aligned with human preferences. We show that instructRL converges to human-like
policies that satisfy the given instructions in a proof-of-concept environment
as well as the challenging Hanabi benchmark. Finally, we show that knowing the
language instruction significantly boosts human-AI coordination performance in
human evaluations in Hanabi.

在缺乏高质量人类行为数据的情况下，使用预训练的大型语言模型生成人类语言指令的先验策略并规范化强化学习目标可以帮助人工智能代理与人类协作，并在多智能体强化学习问题中实现人工智能代理与人类偏好一致的均衡解。案例中验证了该框架的有效性。