For solving zero-sum games involving non-transitivity, a common approach is
to maintain population policies to approximate the Nash Equilibrium (NE).
Previous research has shown that the Policy Space Response Oracle (PSRO) is an
effective multi-agent reinforcement learning framework for these games.
However, repeatedly training new policies from scratch to approximate the Best
Response (BR) to opponents' mixed policies at each iteration is inefficient and
costly. While some PSRO methods initialize a new BR policy by inheriting from
past BR policies, this approach limits the exploration of new policies,
especially against challenging opponents.To address this issue, we propose
Fusion-PSRO, which uses model fusion to initialize the policy for better
approximation to BR. With Top-k probabilities from NE, we select high-quality
base policies and fuse them into a new BR policy through model averaging. This
approach allows the initialized policy to incorporate multiple expert policies,
making it easier to handle difficult opponents compared to inheriting or
initializing from scratch. Additionally, our method only modifies the policy
initialization, enabling its application to nearly all PSRO variants without
additional training overhead.Our experiments with non-transitive matrix games,
Leduc poker, and the more complex Liars Dice demonstrate that Fusion-PSRO
enhances the performance of nearly all PSRO variants, achieving lower
exploitability.

为了解决非传递性的零和游戏问题，该研究提出了一种名为 Fusion-PSRO 的方法，通过模型融合初始化策略，以更好逼近最佳反应策略，并在非传递性矩阵游戏和复杂 Liars Dice 等实验中验证了其在提高几乎所有 PSRO 变体性能方面的有效性。