We propose a novel hierarchical agent architecture for multi-agent reinforcement learning with concealed information. The hierarchy is grounded in the concealed information about other players, which resolves "the chicken or the egg" nature of option discovery. We factorise the value function over a latent representation of the concealed information and then re-use this latent space to factorise the policy into options. Low-level policies (options) are trained to respond to particular states of other agents grouped by the latent representation, while the top level (meta-policy) learns to infer the latent representation from its own observation thereby to select the right option. This grounding facilitates credit assignment across the levels of hierarchy. We show that this helps generalisation---performance against a held-out set of pre-trained competitors, while training in self- or population-play---and resolution of social dilemmas in self-play.

本文研究了多智能体游戏中的泛化问题，提出一种基于博弈论结构的分层代理架构以提高策略推广能力。该方法可以跨层次地进行信用分配，实验结果表明，与传统基线方法相比，分层代理的泛化能力更强。

多智能体强化学习中的行为层次结构建立