In this work, we focus on the task of procedure planning from instructional
videos with text supervision, where a model aims to predict an action sequence
to transform the initial visual state into the goal visual state. A critical
challenge of this task is the large semantic gap between observed visual states
and unobserved intermediate actions, which is ignored by previous works.
Specifically, this semantic gap refers to that the contents in the observed
visual states are semantically different from the elements of some action text
labels in a procedure. To bridge this semantic gap, we propose a novel
event-guided paradigm, which first infers events from the observed states and
then plans out actions based on both the states and predicted events. Our
inspiration comes from that planning a procedure from an instructional video is
to complete a specific event and a specific event usually involves specific
actions. Based on the proposed paradigm, we contribute an Event-guided
Prompting-based Procedure Planning (E3P) model, which encodes event information
into the sequential modeling process to support procedure planning. To further
consider the strong action associations within each event, our E3P adopts a
mask-and-predict approach for relation mining, incorporating a probabilistic
masking scheme for regularization. Extensive experiments on three datasets
demonstrate the effectiveness of our proposed model.

本研究针对从带有文本监督的指令视频中进行过程规划的任务，提出了一种基于事件引导的范式，通过从观察到的状态中推断事件并基于状态和预测的事件规划行动。在提出的模型中，还采用了遮盖和预测方法进行关系挖掘，通过将事件信息编码到顺序建模过程中来支持过程规划。实验证明了我们提出的模型的有效性。