In many real-world applications of control system and robotics, linear
temporal logic (LTL) is a widely-used task specification language which has a
compositional grammar that naturally induces temporally extended behaviours
across tasks, including conditionals and alternative realizations. An important
problem in RL with LTL tasks is to learn task-conditioned policies which can
zero-shot generalize to new LTL instructions not observed in the training.
However, because symbolic observation is often lossy and LTL tasks can have
long time horizon, previous works can suffer from issues such as training
sampling inefficiency and infeasibility or sub-optimality of the found
solutions. In order to tackle these issues, this paper proposes a novel
multi-task RL algorithm with improved learning efficiency and optimality. To
achieve the global optimality of task completion, we propose to learn options
dependent on the future subgoals via a novel off-policy approach. In order to
propagate the rewards of satisfying future subgoals back more efficiently, we
propose to train a multi-step value function conditioned on the subgoal
sequence which is updated with Monte Carlo estimates of multi-step discounted
returns. In experiments on three different domains, we evaluate the LTL
generalization capability of the agent trained by the proposed method, showing
its advantage over previous representative methods.

本文提出了一种新的多任务 RL 算法，通过离线策略学习选项来实现任务完成的全局最优性，并通过训练基于子目标序列的多步价值函数来更有效地传递满足未来子目标的奖励，从而解决了在 RL 中学习 LTL 任务代价函数时的一些问题。实验结果表明，该方法在 LTL 广义化方面具有优势。