Robustness against adversarial attacks and distribution shifts is a
long-standing goal of Reinforcement Learning (RL). To this end, Robust
Adversarial Reinforcement Learning (RARL) trains a protagonist against
destabilizing forces exercised by an adversary in a competitive zero-sum Markov
game, whose optimal solution, i.e., rational strategy, corresponds to a Nash
equilibrium. However, finding Nash equilibria requires facing complex saddle
point optimization problems, which can be prohibitive to solve, especially for
high-dimensional control. In this paper, we propose a novel approach for
adversarial RL based on entropy regularization to ease the complexity of the
saddle point optimization problem. We show that the solution of this
entropy-regularized problem corresponds to a Quantal Response Equilibrium
(QRE), a generalization of Nash equilibria that accounts for bounded
rationality, i.e., agents sometimes play random actions instead of optimal
ones. Crucially, the connection between the entropy-regularized objective and
QRE enables free modulation of the rationality of the agents by simply tuning
the temperature coefficient. We leverage this insight to propose our novel
algorithm, Quantal Adversarial RL (QARL), which gradually increases the
rationality of the adversary in a curriculum fashion until it is fully
rational, easing the complexity of the optimization problem while retaining
robustness. We provide extensive evidence of QARL outperforming RARL and recent
baselines across several MuJoCo locomotion and navigation problems in overall
performance and robustness.

基于熵正则化的量子对抗增强学习 (QARL) 是一种新方法，通过渐进式增加对手的理性来简化优化问题的复杂性，从而提高稳健性以及在多个 MuJoCo 运动和导航问题上的整体性能。