Our goal is to generate a policy to complete an unseen task given just a
single video demonstration of the task in a given domain. We hypothesize that
to successfully generalize to unseen complex tasks from a single video
demonstration, it is necessary to explicitly incorporate the compositional
structure of the tasks into the model. To this end, we propose Neural Task
Graph (NTG) Networks, which use conjugate task graph as the intermediate
representation to modularize both the video demonstration and the derived
policy. We empirically show NTG achieves inter-task generalization on two
complex tasks: Block Stacking in BulletPhysics and Object Collection in
AI2-THOR. NTG improves data efficiency with visual input as well as achieve
strong generalization without the need for dense hierarchical supervision. We
further show that similar performance trends hold when applied to real-world
data. We show that NTG can effectively predict task structure on the JIGSAWS
surgical dataset and generalize to unseen tasks.

通过使用神经任务图网络作为中间表示，我们成功地在两个复杂任务中将任务的组成结构显式融合到模型中，从而实现任务间的泛化，提高数据效率以及无需密集分层监督而实现强泛化能力，并在 JIGSAWS 手术数据集上展示了良好的任务结构预测性能。