Curriculum reinforcement learning (CRL) allows solving complex tasks by
generating a tailored sequence of learning tasks, starting from easy ones and
subsequently increasing their difficulty. Although the potential of curricula
in RL has been clearly shown in various works, it is less clear how to generate
them for a given learning environment, resulting in various methods aiming to
automate this task. In this work, we focus on framing curricula as
interpolations between task distributions, which has previously been shown to
be a viable approach to CRL. Identifying key issues of existing methods, we
frame the generation of a curriculum as a constrained optimal transport problem
between task distributions. Benchmarks show that this way of curriculum
generation can improve upon existing CRL methods, yielding high performance in
various tasks with different characteristics.

通过将课程设置为任务分布之间的插值，将生成课程作为约束优化传输问题来提高课程强化学习（CRL）方法的性能，从而在具有不同特点的各种任务中取得高性能。

关于最优传输在课程强化学习中的益处

On the Benefit of Optimal Transport for Curriculum Reinforcement  Learning

Curriculum Reinforcement Learning (CRL) aims to create a sequence of tasks,
starting from easy ones and gradually learning towards difficult tasks. In this
work, we focus on the idea of framing CRL as interpolations between a source
(auxiliary) and a target task distribution. Although existing studies have
shown the great potential of this idea, it remains unclear how to formally
quantify and generate the movement between task distributions. Inspired by the
insights from gradual domain adaptation in semi-supervised learning, we create
a natural curriculum by breaking down the potentially large task distributional
shift in CRL into smaller shifts. We propose GRADIENT, which formulates CRL as
an optimal transport problem with a tailored distance metric between tasks.
Specifically, we generate a sequence of task distributions as a geodesic
interpolation (i.e., Wasserstein barycenter) between the source and target
distributions. Different from many existing methods, our algorithm considers a
task-dependent contextual distance metric and is capable of handling
nonparametric distributions in both continuous and discrete context settings.
In addition, we theoretically show that GRADIENT enables smooth transfer
between subsequent stages in the curriculum under certain conditions. We
conduct extensive experiments in locomotion and manipulation tasks and show
that our proposed GRADIENT achieves higher performance than baselines in terms
of learning efficiency and asymptotic performance.

该研究将课程机器学习作为优化输运问题来量化和生成任务分布之间的移动，并提出了一种称为 GRADIENT 的算法来处理连续和离散背景下的非参数分布。 实验结果表明，该算法在学习效率和渐进性能方面均优于基线。

利用渐进领域自适应的最优传输进行课程强化学习

Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptation

Meta-learning considers the problem of learning an efficient learning process
that can leverage its past experience to accurately solve new tasks. However,
the efficacy of meta-learning crucially depends on the distribution of tasks
available for training, and this is often assumed to be known a priori or
constructed from limited supervised datasets. In this work, we aim to provide
task distributions for meta-learning by considering self-supervised tasks
automatically proposed from unlabeled text, to enable large-scale meta-learning
in NLP. We design multiple distributions of self-supervised tasks by
considering important aspects of task diversity, difficulty, type, domain, and
curriculum, and investigate how they affect meta-learning performance. Our
analysis shows that all these factors meaningfully alter the task distribution,
some inducing significant improvements in downstream few-shot accuracy of the
meta-learned models. Empirically, results on 20 downstream tasks show
significant improvements in few-shot learning -- adding up to +4.2% absolute
accuracy (on average) to the previous unsupervised meta-learning method, and
perform comparably to supervised methods on the FewRel 2.0 benchmark.

本项研究探讨了如何通过自监督的方式，自动构建和提供能在自然语言处理中进行大规模元学习的任务分布，考虑了任务的多样性、难度、类型、领域和课程，结果表明，这些因素都会有意义地改变任务分布，从而显著提高元学习模型少样本学习的准确度。