Multi-Agent Reinforcement Learning (MARL) has shown promising results across several domains. Despite this promise, MARL policies often lack robustness and are therefore sensitive to small changes in their environment. This presents a serious concern for the real world deployment of MARL algorithms, where the testing environment may slightly differ from the training environment. In this work we show that we can gain robustness by controlling a policy's Lipschitz constant, and under mild conditions, establish the existence of a Lipschitz and close-to-optimal policy. Based on these insights, we propose a new robust MARL framework, ERNIE, that promotes the Lipschitz continuity of the policies with respect to the state observations and actions by adversarial regularization. The ERNIE framework provides robustness against noisy observations, changing transition dynamics, and malicious actions of agents. However, ERNIE's adversarial regularization may introduce some training instability. To reduce this instability, we reformulate adversarial regularization as a Stackelberg game. We demonstrate the effectiveness of the proposed framework with extensive experiments in traffic light control and particle environments. In addition, we extend ERNIE to mean-field MARL with a formulation based on distributionally robust optimization that outperforms its non-robust counterpart and is of independent interest. Our code is available at https://github.com/abukharin3/ERNIE.

通过控制一个策略的利普希茨常数，我们展示了在固定条件下的利普希茨和接近最优策略的存在，并基于此提出了新的强健 MARL 框架 ERNIE，通过对状态观测和动作进行对抗性正则化来提高其的利普希茨连续性，并展示了在交通灯控制和粒子环境中的广泛实验。另外，我们扩展了ERNIE到基于分布鲁棒优化的mean-field MARL，该方法在性能上超越了非强健对照组，并具有独立因素。

通过对抗正则化实现稳健的多智能体强化学习：理论基础和稳定算法