We tackle real-world problems with complex structures beyond the pixel-based
game or simulator. We formulate it as a few-shot reinforcement learning problem
where a task is characterized by a subtask graph that defines a set of subtasks
and their dependencies that are unknown to the agent. Different from the
previous meta-rl methods trying to directly infer the unstructured task
embedding, our multi-task subtask graph inferencer (MTSGI) first infers the
common high-level task structure in terms of the subtask graph from the
training tasks, and use it as a prior to improve the task inference in testing.
Our experiment results on 2D grid-world and complex web navigation domains show
that the proposed method can learn and leverage the common underlying structure
of the tasks for faster adaptation to the unseen tasks than various existing
algorithms such as meta reinforcement learning, hierarchical reinforcement
learning, and other heuristic agents.

本研究提出基于多任务子任务图推断的少样本强化学习方法，通过推断训练任务的高层子任务图结构，提高在测试任务中的任务推断和适应能力，实验结果表明其能够学习并利用任务的潜在共同结构，比其他现有算法如元强化学习、分层强化学习和启发式智能体等更快地适应未见过的任务。

针对少样本任务概括的快速推理和组合任务结构转移

Fast Inference and Transfer of Compositional Task Structures for Few-shot Task Generalization

We introduce a new RL problem where the agent is required to generalize to a
previously-unseen environment characterized by a subtask graph which describes
a set of subtasks and their dependencies. Unlike existing hierarchical
multitask RL approaches that explicitly describe what the agent should do at a
high level, our problem only describes properties of subtasks and relationships
among them, which requires the agent to perform complex reasoning to find the
optimal subtask to execute. To solve this problem, we propose a neural subtask
graph solver (NSGS) which encodes the subtask graph using a recursive neural
network embedding. To overcome the difficulty of training, we propose a novel
non-parametric gradient-based policy, graph reward propagation, to pre-train
our NSGS agent and further finetune it through actor-critic method. The
experimental results on two 2D visual domains show that our agent can perform
complex reasoning to find a near-optimal way of executing the subtask graph and
generalize well to the unseen subtask graphs. In addition, we compare our agent
with a Monte-Carlo tree search (MCTS) method showing that our method is much
more efficient than MCTS, and the performance of NSGS can be further improved
by combining it with MCTS.

提出了一种新的强化学习问题，该问题需要代理在一个由子任务图描述的先前未见的环境中进行泛化，我们提出了一种神经子任务图求解器，通过使用递归神经网络嵌入来编码子任务图，并用基于梯度的策略，图奖励传播来预训练它，进一步通过演员 - 评论家方法微调它，实验结果表明，我们的代理可以进行复杂的推理，找到执行子任务图的近似最优方式，并且在未见的子任务图上具备很好的泛化性能。