In this paper, we explore the capability of an agent to construct a logical
sequence of action steps, thereby assembling a strategic procedural plan. This
plan is crucial for navigating from an initial visual observation to a target
visual outcome, as depicted in real-life instructional videos. Existing works
have attained partial success by extensively leveraging various sources of
information available in the datasets, such as heavy intermediate visual
observations, procedural names, or natural language step-by-step instructions,
for features or supervision signals. However, the task remains formidable due
to the implicit causal constraints in the sequencing of steps and the
variability inherent in multiple feasible plans. To tackle these intricacies
that previous efforts have overlooked, we propose to enhance the capabilities
of the agent by infusing it with procedural knowledge. This knowledge, sourced
from training procedure plans and structured as a directed weighted graph,
equips the agent to better navigate the complexities of step sequencing and its
potential variations. We coin our approach KEPP, a novel Knowledge-Enhanced
Procedure Planning system, which harnesses a probabilistic procedural knowledge
graph extracted from training data, effectively acting as a comprehensive
textbook for the training domain. Experimental evaluations across three
widely-used datasets under settings of varying complexity reveal that KEPP
attains superior, state-of-the-art results while requiring only minimal
supervision.

该研究探讨了代理程序在构建逻辑动作步骤序列方面的能力，从而组装战略性的过程计划。利用来自训练数据的程序化知识图，我们提出了一种名为 KEPP 的新型知识增强程序规划系统，该系统在各种复杂度设置下的实验评估结果表明，KEPP 取得了卓越的、最新的结果，同时只需要很少的监督。