We consider the problem of curriculum design for reinforcement learning (RL) agents in contextual multi-task settings. Existing techniques on automatic curriculum design typically require domain-specific hyperparameter tuning or have limited theoretical underpinnings. To tackle these limitations, we design our curriculum strategy, ProCuRL, inspired by the pedagogical concept of Zone of Proximal Development (ZPD). ProCuRL captures the intuition that learning progress is maximized when picking tasks that are neither too hard nor too easy for the learner. We mathematically derive ProCuRL by analyzing two simple learning settings. We also present a practical variant of ProCuRL that can be directly integrated with deep RL frameworks with minimal hyperparameter tuning. Experimental results on a variety of domains demonstrate the effectiveness of our curriculum strategy over state-of-the-art baselines in accelerating the training process of deep RL agents.

通过分析两个简单的学习场景，我们设计了一种叫做 ProCuRL 的课程策略，受到了教育概念中的“近发展区”（Zone of Proximal Development，ZPD）的启发。我们还提出了 ProCuRL 的实用变体，可在最小的超参数调整下直接与深度强化学习框架集成，从而有效地加速深度强化学习代理的训练过程。实验结果表明，与现有的基线相比，我们的课程策略在各种领域中都取得了较好的效果。

强化学习代理的近端课程