Self-interested individuals often fail to cooperate, posing a fundamental challenge for multi-agent learning. How can we achieve cooperation among self-interested, independent learning agents? Promising recent work has shown that in certain tasks cooperation can be established between learning-aware agents who model the learning dynamics of each other. Here, we present the first unbiased, higher-derivative-free policy gradient algorithm for learning-aware reinforcement learning, which takes into account that other agents are themselves learning through trial and error based on multiple noisy trials. We then leverage efficient sequence models to condition behavior on long observation histories that contain traces of the learning dynamics of other agents. Training long-context policies with our algorithm leads to cooperative behavior and high returns on standard social dilemmas, including a challenging environment where temporally-extended action coordination is required. Finally, we derive from the iterated prisoner's dilemma a novel explanation for how and when cooperation arises among self-interested learning-aware agents.

本文研究了在自利的独立学习体之间实现合作的挑战，并提出了一种首个无偏高阶无梯度的策略梯度算法，专注于学习感知的强化学习。通过利用高效的序列模型，我们的算法能够在包含其他智能体学习动态的长观测历史上调节行为，从而在标准社交困境中实现合作行为和高回报。

通过学习感知的策略梯度实现多智能体合作