Learning from Demonstrations (LfD) via Behavior Cloning (BC) works well on
multiple complex tasks. However, a limitation of the typical LfD approach is
that it requires expert demonstrations for all scenarios, including those in
which the algorithm is already well-trained. The recently proposed Learning
from Interventions (LfI) overcomes this limitation by using an expert overseer.
The expert overseer only intervenes when it suspects that an unsafe action is
about to be taken. Although LfI significantly improves over LfD, the
state-of-the-art LfI fails to account for delay caused by the expert's reaction
time and only learns short-term behavior. We address these limitations by 1)
interpolating the expert's interventions back in time, and 2) by splitting the
policy into two hierarchical levels, one that generates sub-goals for the
future and another that generates actions to reach those desired sub-goals.
This sub-goal prediction forces the algorithm to learn long-term behavior while
also being robust to the expert's reaction time. Our experiments show that LfI
using sub-goals in a hierarchical policy framework trains faster and achieves
better asymptotic performance than typical LfD.

通过将专家的干预向前插值并将策略分为两个层次，一个生成未来的子目标，另一个生成到达这些期望子目标的动作，我们在学习过程中引入了目标预测的思想，从而增强算法学习长期行为，同时还能够抵御专家反应时间的影响，提高了机器人上的学习效果。