A significant roadblock to the development of principled multi-agent
reinforcement learning is the fact that desired solution concepts like Nash
equilibria may be intractable to compute. To overcome this obstacle, we take
inspiration from behavioral economics and show that -- by imbuing agents with
important features of human decision-making like risk aversion and bounded
rationality -- a class of risk-averse quantal response equilibria (RQE) become
tractable to compute in all $n$-player matrix and finite-horizon Markov games.
In particular, we show that they emerge as the endpoint of no-regret learning
in suitably adjusted versions of the games. Crucially, the class of
computationally tractable RQE is independent of the underlying game structure
and only depends on agents' degree of risk-aversion and bounded rationality. To
validate the richness of this class of solution concepts we show that it
captures peoples' patterns of play in a number of 2-player matrix games
previously studied in experimental economics. Furthermore, we give a first
analysis of the sample complexity of computing these equilibria in
finite-horizon Markov games when one has access to a generative model and
validate our findings on a simple multi-agent reinforcement learning benchmark.

通过赋予智能体风险厌恶和有限理性等人类决策要素，我们展示了一类风险厌恶量子响应均衡解（Risk-Averse Quantal Response Equilibria，RQE），不依赖于底层游戏结构而只依赖于智能体的风险厌恶程度和有限理性，在所有 n 个玩家矩阵和有限时域马尔可夫博弈中可以高效计算。此外，我们还通过实证经济学研究所涉及的许多两人矩阵游戏验证了这类解集的丰富性，并对在有生成模型的有限时域马尔可夫博弈中计算这些均衡的样本复杂度进行了首次分析，同时在简单的多智能体强化学习基准测试中验证了我们的发现。

马尔可夫博弈中应对风险偏好的易处理均衡计算

Tractable Equilibrium Computation in Markov Games through Risk Aversion

This work studies an algorithm, which we call magnetic mirror descent, that
is inspired by mirror descent and the non-Euclidean proximal gradient
algorithm. Our contribution is demonstrating the virtues of magnetic mirror
descent as both an equilibrium solver and as an approach to reinforcement
learning in two-player zero-sum games. These virtues include: 1) Being the
first quantal response equilibria solver to achieve linear convergence for
extensive-form games with first order feedback; 2) Being the first standard
reinforcement learning algorithm to achieve empirically competitive results
with CFR in tabular settings; 3) Achieving favorable performance in 3x3 Dark
Hex and Phantom Tic-Tac-Toe as a self-play deep reinforcement learning
algorithm.

研究磁镜下降算法作为均衡求解器和两人零和游戏强化学习的方法，并证明其在多种场景中都可以取得优秀的性能表现，包括在可扩展形式下的均衡求解和在表格设置下的竞争性结果与 CFR 相比。