This paper investigates the model-based methods in multi-agent reinforcement learning (MARL). We specify the dynamics sample complexity and the opponent sample complexity in MARL, and conduct a theoretic analysis of return discrepancy upper bound. To reduce the upper bound with the intention of low sample complexity during the whole learning process, we propose a novel decentralized model-based MARL method, named Adaptive Opponent-wise Rollout Policy Optimization (AORPO). In AORPO, each agent builds its multi-agent environment model, consisting of a dynamics model and multiple opponent models, and trains its policy with the adaptive opponent-wise rollout. We further prove the theoretic convergence of AORPO under reasonable assumptions. Empirical experiments on competitive and cooperative tasks demonstrate that AORPO can achieve improved sample efficiency with comparable asymptotic performance over the compared MARL methods.

本文研究了多智能体强化学习中的基于模型的方法，提出了自适应对手推演策略优化（AORPO）的分散化的基于模型的MARL方法，旨在全过程中降低样本复杂度，并在竞争和合作任务中开展实证研究，表明AORPO能够实现比对比MARL方法更好的样本效率和相似的渐近性能。

基于模型的多智能体策略优化与对手智能化决策