We initiate the study of Multi-Agent Reinforcement Learning from Human Feedback (MARLHF), exploring both theoretical foundations and empirical validations. We define the task as identifying Nash equilibrium from a preference-only offline dataset in general-sum games, a problem marked by the challenge of sparse feedback signals. Our theory establishes the upper complexity bounds for Nash Equilibrium in effective MARLHF, demonstrating that single-policy coverage is inadequate and highlighting the importance of unilateral dataset coverage. These theoretical insights are verified through comprehensive experiments. To enhance the practical performance, we further introduce two algorithmic techniques. (1) We propose a Mean Squared Error (MSE) regularization along the time axis to achieve a more uniform reward distribution and improve reward learning outcomes. (2) We utilize imitation learning to approximate the reference policy, ensuring stability and effectiveness in training. Our findings underscore the multifaceted approach required for MARLHF, paving the way for effective preference-based multi-agent systems.

本研究解决了基于人类反馈的多智能体强化学习（MARLHF）的理论和实证基础，特别是从偏好数据集中识别纳什均衡的问题。通过确定单一策略覆盖不足的理论界限，强调了单方面数据集覆盖的重要性，并提出了时间轴上的均方误差正则化和模仿学习等算法技术，以提升实际性能。研究结果为基于偏好的多智能体系统的有效性奠定了基础。

基于人类反馈的多智能体强化学习：数据覆盖与算法技术