This paper investigates generalisation in multi-agent games, where the
generality of the agent can be evaluated by playing against opponents it hasn't
seen during training. We propose two new games with concealed information and
complex, non-transitive reward structure (think rock/paper/scissors). It turns
out that most current deep reinforcement learning methods fail to efficiently
explore the strategy space, thus learning policies that generalise poorly to
unseen opponents. We then propose a novel hierarchical agent architecture,
where the hierarchy is grounded in the game-theoretic structure of the game --
the top level chooses strategic responses to opponents, while the low level
implements them into policy over primitive actions. This grounding facilitates
credit assignment across the levels of hierarchy. Our experiments show that the
proposed hierarchical agent is capable of generalisation to unseen opponents,
while conventional baselines fail to generalise whatsoever.

本文研究了多智能体游戏中的泛化问题，提出一种基于博弈论结构的分层代理架构以提高策略推广能力。该方法可以跨层次地进行信用分配，实验结果表明，与传统基线方法相比，分层代理的泛化能力更强。