This paper introduces Fast Linearized Adaptive Policy (FLAP), a new meta-reinforcement learning (meta-RL) method that is able to extrapolate well to out-of-distribution tasks without the need to reuse data from training, and adapt almost instantaneously with the need of only a few samples during testing. FLAP builds upon the idea of learning a shared linear representation of the policy so that when adapting to a new task, it suffices to predict a set of linear weights. A separate adapter network is trained simultaneously with the policy such that during adaptation, we can directly use the adapter network to predict these linear weights instead of updating a meta-policy via gradient descent, such as in prior meta-RL methods like MAML, to obtain the new policy. The application of the separate feed-forward network not only speeds up the adaptation run-time significantly, but also generalizes extremely well to very different tasks that prior Meta-RL methods fail to generalize to. Experiments on standard continuous-control meta-RL benchmarks show FLAP presenting significantly stronger performance on out-of-distribution tasks with up to double the average return and up to 8X faster adaptation run-time speeds when compared to prior methods.

本文介绍了一种新的元强化学习方法：Fast Linearized Adaptive Policy（FLAP），它无需重复使用训练数据即能良好推广到不同任务，同时只需在测试期间少量样本即可相当迅速地适应任务。FLAP的主要思想是学习一个策略的共享线性表示，以便适应新任务时只需预测一组线性权重；同时训练一个适配器网络，以便在适应过程中可以直接使用适配器网络预测这些线性权重，从而无需通过梯度下降更新一个元策略以获得新策略。该文结果表明，和之前的Meta-RL方法相比，在标准连续控制元RL基准测试中，FLAP在处理不同任务时表现出显著的性能优势，包括在处理上述任务时可达到的两倍平均回报和最多快8倍的适应运行时间。

基于线性表示的元强化学习用于快速适应