Recent progress in reinforcement learning (RL) has started producing
generally capable agents that can solve a distribution of complex environments.
These agents are typically tested on fixed, human-authored environments. On the
other hand, quality diversity (QD) optimization has been proven to be an
effective component of environment generation algorithms, which can generate
collections of high-quality environments that are diverse in the resulting
agent behaviors. However, these algorithms require potentially expensive
simulations of agents on newly generated environments. We propose Deep
Surrogate Assisted Generation of Environments (DSAGE), a sample-efficient QD
environment generation algorithm that maintains a deep surrogate model for
predicting agent behaviors in new environments. Results in two benchmark
domains show that DSAGE significantly outperforms existing QD environment
generation algorithms in discovering collections of environments that elicit
diverse behaviors of a state-of-the-art RL agent and a planning agent. Our
source code and videos are available at this https URL.

本论文介绍了一种深度代理辅助环境生成技术（DSAGE）, 该技术可以有效生成高质量、多样化的环境，以探索最先进的强化学习和规划代理的多样行为。

基于深度代理的环境生成

Deep Surrogate Assisted Generation of Environments

Many real-world problems are compositional - solving them requires completing
interdependent sub-tasks, either in series or in parallel, that can be
represented as a dependency graph. Deep reinforcement learning (RL) agents
often struggle to learn such complex tasks due to the long time horizons and
sparse rewards. To address this problem, we present Compositional Design of
Environments (CoDE), which trains a Generator agent to automatically build a
series of compositional tasks tailored to the RL agent's current skill level.
This automatic curriculum not only enables the agent to learn more complex
tasks than it could have otherwise, but also selects tasks where the agent's
performance is weak, enhancing its robustness and ability to generalize
zero-shot to unseen tasks at test-time. We analyze why current environment
generation techniques are insufficient for the problem of generating
compositional tasks, and propose a new algorithm that addresses these issues.
Our results assess learning and generalization across multiple compositional
tasks, including the real-world problem of learning to navigate and interact
with web pages. We learn to generate environments composed of multiple pages or
rooms, and train RL agents capable of completing wide-range of complex tasks in
those environments. We contribute two new benchmark frameworks for generating
compositional tasks, compositional MiniGrid and gMiniWoB for web
navigation.CoDE yields 4x higher success rate than the strongest baseline, and
demonstrates strong performance of real websites learned on 3500 primitive
tasks.

通过训练生成器创建任务序列的自动课程表，以使强化学习代理能够完成复杂的组合任务，提高其健壮性和泛化能力，我们提出了一种名为 CoDE 的新算法，并对其进行了分析和评估

零样本组合强化学习环境生成

Environment Generation for Zero-Shot Compositional Reinforcement Learning

A wide range of reinforcement learning (RL) problems - including robustness,
transfer learning, unsupervised RL, and emergent complexity - require
specifying a distribution of tasks or environments in which a policy will be
trained. However, creating a useful distribution of environments is error
prone, and takes a significant amount of developer time and effort. We propose
Unsupervised Environment Design (UED) as an alternative paradigm, where
developers provide environments with unknown parameters, and these parameters
are used to automatically produce a distribution over valid, solvable
environments. Existing approaches to automatically generating environments
suffer from common failure modes: domain randomization cannot generate
structure or adapt the difficulty of the environment to the agent's learning
progress, and minimax adversarial training leads to worst-case environments
that are often unsolvable. To generate structured, solvable environments for
our protagonist agent, we introduce a second, antagonist agent that is allied
with the environment-generating adversary. The adversary is motivated to
generate environments which maximize regret, defined as the difference between
the protagonist and antagonist agent's return. We call our technique
Protagonist Antagonist Induced Regret Environment Design (PAIRED). Our
experiments demonstrate that PAIRED produces a natural curriculum of
increasingly complex environments, and PAIRED agents achieve higher zero-shot
transfer performance when tested in highly novel environments.

本文提出一种名为 “PAIRED” 的技术，可自动设计一系列结构化的智能环境用于训练机器学习模型，实验证明 PAIRED 相较于传统方法在零样本迁移学习及新颖环境下的测试表现更优。