Training AI with strong and rich strategies in multi-agent environments
remains an important research topic in Deep Reinforcement Learning (DRL). The
AI's strength is closely related to its diversity of strategies, and this
relationship can guide us to train AI with both strong and rich strategies. To
prove this point, we propose Diversity is Strength (DIS), a novel DRL training
framework that can simultaneously train multiple kinds of AIs. These AIs are
linked through an interconnected history model pool structure, which enhances
their capabilities and strategy diversities. We also design a model evaluation
and screening scheme to select the best models to enrich the model pool and
obtain the final AI. The proposed training method provides diverse,
generalizable, and strong AI strategies without using human data. We tested our
method in an AI competition based on Google Research Football (GRF) and won the
5v5 and 11v11 tracks. The method enables a GRF AI to have a high level on both
5v5 and 11v11 tracks for the first time, which are under complex multi-agent
environments. The behavior analysis shows that the trained AI has rich
strategies, and the ablation experiments proved that the designed modules
benefit the training process.

本研究提出了「多样性即实力（DIS）」深度强化学习训练框架，该框架可以同时训练多种类型的人工智能，以增强其能力和策略多样性，并通过模型评估和筛选方案选择最佳模型以丰富模型池并获得最终的人工智能。该方法通过在 Google Research Football（GRF）基础上进行的人工智能比赛的测试，在复杂的多智能体环境中首次使 GRF 人工智能在 5v5 和 11v11 赛道上都有很高的水平，其行为分析表明所训练的人工智能具有丰富的策略，并且消融实验证明了设计的模块有益于训练过程。