Open-ended learning methods that automatically generate a curriculum of
increasingly challenging tasks serve as a promising avenue toward generally
capable reinforcement learning agents. Existing methods adapt curricula
independently over either environment parameters (in single-agent settings) or
co-player policies (in multi-agent settings). However, the strengths and
weaknesses of co-players can manifest themselves differently depending on
environmental features. It is thus crucial to consider the dependency between
the environment and co-player when shaping a curriculum in multi-agent domains.
In this work, we use this insight and extend Unsupervised Environment Design
(UED) to multi-agent environments. We then introduce Multi-Agent Environment
Design Strategist for Open-Ended Learning (MAESTRO), the first multi-agent UED
approach for two-player zero-sum settings. MAESTRO efficiently produces
adversarial, joint curricula over both environments and co-players and attains
minimax-regret guarantees at Nash equilibrium. Our experiments show that
MAESTRO outperforms a number of strong baselines on competitive two-player
games, spanning discrete and continuous control settings.

本文提出了 Multi-Agent Environment Design Strategist for Open-Ended Learning (MAESTRO) 算法，该算法在多智能体环境中使用 Unsupervised Environment Design (UED) 生成敌对性的联合任务集，MAESTRO 在竞争性的两人游戏中，跨离散和连续控制环境优于其他基线方法，并在 Nash 均衡点上获得了最小化后悔的保证。

MAESTRO: 多智能体强化学习的开放环境设计

MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement  Learning

We are interested in training general-purpose reinforcement learning agents
that can solve a wide variety of goals. Training such agents efficiently
requires automatic generation of a goal curriculum. This is challenging as it
requires (a) exploring goals of increasing difficulty, while ensuring that the
agent (b) is exposed to a diverse set of goals in a sample efficient manner and
(c) does not catastrophically forget previously solved goals. We propose
Curriculum Self Play (CuSP), an automated goal generation framework that seeks
to satisfy these desiderata by virtue of a multi-player game with four agents.
We extend the asymmetric curricula learning in PAIRED (Dennis et al., 2020) to
a symmetrized game that carefully balances cooperation and competition between
two off-policy student learners and two regret-maximizing teachers. CuSP
additionally introduces entropic goal coverage and accounts for the
non-stationary nature of the students, allowing us to automatically induce a
curriculum that balances progressive exploration with anti-catastrophic
exploitation. We demonstrate that our method succeeds at generating an
effective curricula of goals for a range of control tasks, outperforming other
methods at zero-shot test-time generalization to novel out-of-distribution
goals.

本研究提出一种自动化生成目标课程的框架 ——Curriculum Self Play（CuSP），通过多人游戏的方式平衡合作和竞争的关系，解决了训练通用强化学习智能体所面临的困难，并成功地生成了适用于各种控制任务的目标课程。