We consider the problem of curriculum design for reinforcement learning (RL)
agents in contextual multi-task settings. Existing techniques on automatic
curriculum design typically require domain-specific hyperparameter tuning or
have limited theoretical underpinnings. To tackle these limitations, we design
our curriculum strategy, ProCuRL, inspired by the pedagogical concept of Zone
of Proximal Development (ZPD). ProCuRL captures the intuition that learning
progress is maximized when picking tasks that are neither too hard nor too easy
for the learner. We mathematically derive ProCuRL by analyzing two simple
learning settings. We also present a practical variant of ProCuRL that can be
directly integrated with deep RL frameworks with minimal hyperparameter tuning.
Experimental results on a variety of domains demonstrate the effectiveness of
our curriculum strategy over state-of-the-art baselines in accelerating the
training process of deep RL agents.

通过分析两个简单的学习场景，我们设计了一种叫做 ProCuRL 的课程策略，受到了教育概念中的 “近发展区”（Zone of Proximal Development，ZPD）的启发。我们还提出了 ProCuRL 的实用变体，可在最小的超参数调整下直接与深度强化学习框架集成，从而有效地加速深度强化学习代理的训练过程。实验结果表明，与现有的基线相比，我们的课程策略在各种领域中都取得了较好的效果。