The interplay between exploration and exploitation in competitive multi-agent learning is still far from being well understood. Motivated by this, we study smooth Q-learning, a prototypical learning model that explicitly captures the balance between game rewards and exploration costs. We show that Q-learning always converges to the unique quantal-response equilibrium (QRE), the standard solution concept for games under bounded rationality, in weighted zero-sum polymatrix games with heterogeneous learning agents using positive exploration rates. Complementing recent results about convergence in weighted potential games, we show that fast convergence of Q-learning in competitive settings is obtained regardless of the number of agents and without any need for parameter fine-tuning. As showcased by our experiments in network zero-sum games, these theoretical results provide the necessary guarantees for an algorithmic approach to the currently open problem of equilibrium selection in competitive multi-agent settings.

通过研究充分考虑游戏奖励和探索成本平衡的原型学习模型Q-learning，我们证明在使用积极探索率的具有异质性学习代理的权重零和多元矩阵游戏中，Q-learning总是收敛于唯一的量刑-反应均衡（QRE），这是有界理性下游戏的标准解决方案概念，并展示了 Q-learning 在竞争环境中的快速收敛性，而无需任何参数微调，在竞争多代理环境中的均衡选择问题提供了算法所需的保证。

多智能体竞赛中的探索-利用：有界理性下的收敛