This paper investigates a population-based training regime based on
game-theoretic principles called Policy-Spaced Response Oracles (PSRO). PSRO is
general in the sense that it (1) encompasses well-known algorithms such as
fictitious play and double oracle as special cases, and (2) in principle
applies to general-sum, many-player games. Despite this, prior studies of PSRO
have been focused on two-player zero-sum games, a regime wherein Nash
equilibria are tractably computable. In moving from two-player zero-sum games
to more general settings, computation of Nash equilibria quickly becomes
infeasible. Here, we extend the theoretical underpinnings of PSRO by
considering an alternative solution concept, $\alpha$-Rank, which is unique
(thus faces no equilibrium selection issues, unlike Nash) and applies readily
to general-sum, many-player settings. We establish convergence guarantees in
several games classes, and identify links between Nash equilibria and
$\alpha$-Rank. We demonstrate the competitive performance of
$\alpha$-Rank-based PSRO against an exact Nash solver-based PSRO in 2-player
Kuhn and Leduc Poker. We then go beyond the reach of prior PSRO applications by
considering 3- to 5-player poker games, yielding instances where $\alpha$-Rank
achieves faster convergence than approximate Nash solvers, thus establishing it
as a favorable general games solver. We also carry out an initial empirical
validation in MuJoCo soccer, illustrating the feasibility of the proposed
approach in another complex domain.

本文基于博弈论原理研究了一种基于人口统计的培训体系 —— 策略空间响应神谕（PSRO），并将其扩展到广义和多人游戏中。通过使用另一种解决方案概念 $\alpha$-Rank，在一些游戏分类中建立了收敛保证，并确定了 Nash 平衡和 $\alpha$-Rank 之间的联系。实验结果表明，基于 $\alpha$-Rank 的 PSRO 可以在很多游戏中实现比近似 Nash Solver 更快的收敛速度。