Online resources such as WikiHow compile a wide range of scripts for
performing everyday tasks, which can assist models in learning to reason about
procedures. However, the scripts are always presented in a linear manner, which
does not reflect the flexibility displayed by people executing tasks in real
life. For example, in the CrossTask Dataset, 64.5% of consecutive step pairs
are also observed in the reverse order, suggesting their ordering is not fixed.
In addition, each step has an average of 2.56 frequent next steps,
demonstrating "branching". In this paper, we propose the new challenging task
of non-sequential graph script induction, aiming to capture optional and
interchangeable steps in procedural planning. To automate the induction of such
graph scripts for given tasks, we propose to take advantage of loosely aligned
videos of people performing the tasks. In particular, we design a multimodal
framework to ground procedural videos to WikiHow textual steps and thus
transform each video into an observed step path on the latent ground truth
graph script. This key transformation enables us to train a script knowledge
model capable of both generating explicit graph scripts for learnt tasks and
predicting future steps given a partial step sequence. Our best model
outperforms the strongest pure text/vision baselines by 17.52% absolute gains
on F1@3 for next step prediction and 13.8% absolute gains on Acc@1 for partial
sequence completion. Human evaluation shows our model outperforming the WikiHow
linear baseline by 48.76% absolute gains in capturing sequential and
non-sequential step relationships.

该论文提出了一种新的具有挑战性的任务，即非顺序图脚本推理，旨在捕捉程序规划中的可选步骤和可互换步骤，通过多模态框架将视频转换成观察到的步骤路径，以训练生成明确图脚本和预测未来步骤的模型。该模型优于纯文本 / 视觉基线，人类评估显示超过 WikiHow 线性基线，能够更好地捕捉顺序和非顺序步骤关系。