Optimizing a set of functions simultaneously by leveraging their similarity
is called multi-task optimization. Current black-box multi-task algorithms only
solve a finite set of tasks, even when the tasks originate from a continuous
space. In this paper, we introduce Parametric-task MAP-Elites (PT-ME), a novel
black-box algorithm to solve continuous multi-task optimization problems. This
algorithm (1) solves a new task at each iteration, effectively covering the
continuous space, and (2) exploits a new variation operator based on local
linear regression. The resulting dataset of solutions makes it possible to
create a function that maps any task parameter to its optimal solution. We show
on two parametric-task toy problems and a more realistic and challenging
robotic problem in simulation that PT-ME outperforms all baselines, including
the deep reinforcement learning algorithm PPO.

利用相似性同时优化一组函数称为多任务优化。本文介绍参数化任务 MAP-Elites（PT-ME），一种解决连续多任务优化问题的新型黑盒算法。该算法每次迭代解决一个新任务，从而有效地覆盖连续空间，并利用基于局部线性回归的新型变异操作。在两个参数化任务玩具问题和一个更现实和具有挑战性的仿真机器人问题中，我们展示了 PT-ME 优于包括深度强化学习算法 PPO 在内的所有对照算法。

参数化任务的 MAP-Elites

Parametric-Task MAP-Elites

The most data-efficient algorithms for reinforcement learning (RL) in
robotics are based on uncertain dynamical models: after each episode, they
first learn a dynamical model of the robot, then they use an optimization
algorithm to find a policy that maximizes the expected return given the model
and its uncertainties. It is often believed that this optimization can be
tractable only if analytical, gradient-based algorithms are used; however,
these algorithms require using specific families of reward functions and
policies, which greatly limits the flexibility of the overall approach. In this
paper, we introduce a novel model-based RL algorithm, called Black-DROPS
(Black-box Data-efficient RObot Policy Search) that: (1) does not impose any
constraint on the reward function or the policy (they are treated as
black-boxes), (2) is as data-efficient as the state-of-the-art algorithm for
data-efficient RL in robotics, and (3) is as fast (or faster) than analytical
approaches when several cores are available. The key idea is to replace the
gradient-based optimization algorithm with a parallel, black-box algorithm that
takes into account the model uncertainties. We demonstrate the performance of
our new algorithm on two standard control benchmark problems (in simulation)
and a low-cost robotic manipulator (with a real robot).

本文提出一种称为 Black-DROPS 的新型基于模型的强化学习算法，使用黑盒算法代替基于梯度的优化算法，可以更灵活地处理奖励函数和策略，并且在数据效率方面与现有最先进算法相当。在模拟环境和机器人系统上进行验证。