Planning with a learned model is arguably a key component of intelligence. There are several challenges in realizing such a component in large-scale reinforcement learning (RL) problems. One such challenge is dealing effectively with continuous action spaces when using tree-search planning (e.g., it is not feasible to consider every action even at just the root node of the tree). In this paper we present a method for selecting affordances useful for planning -- for learning which small number of actions/options from a continuous space of actions/options to consider in the tree-expansion process during planning. We consider affordances that are goal-and-state-conditional mappings to actions/options as well as unconditional affordances that simply select actions/options available in all states. Our selection method is gradient based: we compute gradients through the planning procedure to update the parameters of the function that represents affordances. Our empirical work shows that it is feasible to learn to select both primitive-action and option affordances, and that simultaneously learning to select affordances and planning with a learned value-equivalent model can outperform model-free RL.

本篇论文主要探讨在大规模强化学习领域中，如何解决在使用树搜索规划时处理连续动作空间的问题，并通过学习选取能够有助于规划的可接受行为（Affordances），并以基于梯度下降的方法更新其参数，从而实现同时学习选取单元行为和规划带有学习后价值等价模型的方法优于无模型强化学习的目的。

GrASP: 基于梯度的可供性选择规划