In multi-agent settings with mixed incentives, methods developed for zero-sum
games have been shown to lead to detrimental outcomes. To address this issue,
opponent shaping (OS) methods explicitly learn to influence the learning
dynamics of co-players and empirically lead to improved individual and
collective outcomes. However, OS methods have only been evaluated in
low-dimensional environments due to the challenges associated with estimating
higher-order derivatives or scaling model-free meta-learning. Alternative
methods that scale to more complex settings either converge to undesirable
solutions or rely on unrealistic assumptions about the environment or
co-players. In this paper, we successfully scale an OS-based approach to
general-sum games with temporally-extended actions and long-time horizons for
the first time. After analysing the representations of the meta-state and
history used by previous algorithms, we propose a simplified version called
Shaper. We show empirically that Shaper leads to improved individual and
collective outcomes in a range of challenging settings from literature. We
further formalize a technique previously implicit in the literature, and
analyse its contribution to opponent shaping. We show empirically that this
technique is helpful for the functioning of prior methods in certain
environments. Lastly, we show that previous environments, such as the CoinGame,
are inadequate for analysing temporally-extended general-sum interactions.

对于混合激励的多智能体环境中，通过学习对博弈对手产生影响的对手塑造方法，我们成功将其扩展到具有长期行动和长期视角的广义和博弈，提出了一个称为 Shaper 的简化版本，并证明 Shaper 在多种具有挑战性的环境中能够改善个体和整体的结果。