Zero-sum games have long guided artificial intelligence research, since they
possess both a rich strategy space of best-responses and a clear evaluation
metric. What's more, competition is a vital mechanism in many real-world
multi-agent systems capable of generating intelligent innovations: Darwinian
evolution, the market economy and the AlphaZero algorithm, to name a few. In
two-player zero-sum games, the challenge is usually viewed as finding Nash
equilibrium strategies, safeguarding against exploitation regardless of the
opponent. While this captures the intricacies of chess or Go, it avoids the
notion of cooperation with co-players, a hallmark of the major transitions
leading from unicellular organisms to human civilization. Beyond two players,
alliance formation often confers an advantage; however this requires trust,
namely the promise of mutual cooperation in the face of incentives to defect.
Successful play therefore requires adaptation to co-players rather than the
pursuit of non-exploitability. Here we argue that a systematic study of
many-player zero-sum games is a crucial element of artificial intelligence
research. Using symmetric zero-sum matrix games, we demonstrate formally that
alliance formation may be seen as a social dilemma, and empirically that
na\"ive multi-agent reinforcement learning therefore fails to form alliances.
We introduce a toy model of economic competition, and show how reinforcement
learning may be augmented with a peer-to-peer contract mechanism to discover
and enforce alliances. Finally, we generalize our agent model to incorporate
temporally-extended contracts, presenting opportunities for further work.

本研究通过对称的零和矩阵游戏，证明了联盟形成可以被看作是一个社会困境，并且实证表明，在多代理强化学习中，天真的方法往往失败。同时，我们引入了点对点的合约机制来发现和执行联盟。最后，我们将我们的代理模型推广到了包括时间延伸合约的情况，并提出了进一步研究的机会。