Given the enormous number of instructional videos available online, learning
a diverse array of multi-step task models from videos is an appealing goal. We
introduce a new pre-trained video model, VideoTaskformer, focused on
representing the semantics and structure of instructional videos. We pre-train
VideoTaskformer using a simple and effective objective: predicting weakly
supervised textual labels for steps that are randomly masked out from an
instructional video (masked step modeling). Compared to prior work which learns
step representations locally, our approach involves learning them globally,
leveraging video of the entire surrounding task as context. From these learned
representations, we can verify if an unseen video correctly executes a given
task, as well as forecast which steps are likely to be taken after a given
step. We introduce two new benchmarks for detecting mistakes in instructional
videos, to verify if there is an anomalous step and if steps are executed in
the right order. We also introduce a long-term forecasting benchmark, where the
goal is to predict long-range future steps from a given step. Our method
outperforms previous baselines on these tasks, and we believe the tasks will be
a valuable way for the community to measure the quality of step
representations. Additionally, we evaluate VideoTaskformer on 3 existing
benchmarks -- procedural activity recognition, step classification, and step
forecasting -- and demonstrate on each that our method outperforms existing
baselines and achieves new state-of-the-art performance.

本论文介绍了一种新的预训练视频模型，VideoTaskformer，利用遮盖步骤建立弱监督学习任务，学习全局步骤表示，实现对多步骤任务的表示和预测。同时提出了两个新的基准测试来检测指令视频中的错误步骤和步骤执行顺序，以及一项长期预测基准测试，均取得了优于现有基线的性能，适用于评估全球学习的步骤表示质量。