Procedural videos show step-by-step demonstrations of tasks like recipe
preparation. Understanding such videos is challenging, involving the precise
localization of steps and the generation of textual instructions. Manually
annotating steps and writing instructions is costly, which limits the size of
current datasets and hinders effective learning. Leveraging large but noisy
video-transcript datasets for pre-training can boost performance, but demands
significant computational resources. Furthermore, transcripts contain
irrelevant content and exhibit style variation compared to instructions written
by human annotators. To mitigate both issues, we propose a technique,
Sieve-&-Swap, to automatically curate a smaller dataset: (i) Sieve filters
irrelevant transcripts and (ii) Swap enhances the quality of the text
instruction by automatically replacing the transcripts with human-written
instructions from a text-only recipe dataset. The curated dataset, three orders
of magnitude smaller than current web-scale datasets, enables efficient
training of large-scale models with competitive performance. We complement our
Sieve-\&-Swap approach with a Procedure Transformer (ProcX) for end-to-end step
localization and instruction generation for procedural videos. When this model
is pre-trained on our curated dataset, it achieves state-of-the-art performance
in zero-shot and finetuning settings on YouCook2 and Tasty, while using a
fraction of the computational resources.

通过 Sieve 和 Swap 技术自动筛选和替换人工写入的文本指导，构建一个比当前大规模数据集小三个数量级的精选数据集，从而提高大规模模型在过程视频中的性能。同时使用 Procedure Transformer (ProcX) 进行步骤定位和指导生成，以零 - shot 和微调设置在 YouCook2 和 Tasty 数据集上取得了最先进的性能，并且使用了较少的计算资源。

视频本地化指令生成的高效预训练

Efficient Pre-training for Localized Instruction Generation of Videos

We study the problem of future step anticipation in procedural videos. Given
a video of an ongoing procedural activity, we predict a plausible next
procedure step described in rich natural language. While most previous work
focus on the problem of data scarcity in procedural video datasets, another
core challenge of future anticipation is how to account for multiple plausible
future realizations in natural settings. This problem has been largely
overlooked in previous work. To address this challenge, we frame future step
prediction as modelling the distribution of all possible candidates for the
next step. Specifically, we design a generative model that takes a series of
video clips as input, and generates multiple plausible and diverse candidates
(in natural language) for the next step. Following previous work, we side-step
the video annotation scarcity by pretraining our model on a large text-based
corpus of procedural activities, and then transfer the model to the video
domain. Our experiments, both in textual and video domains, show that our model
captures diversity in the next step prediction and generates multiple plausible
future predictions. Moreover, our model establishes new state-of-the-art
results on YouCookII, where it outperforms existing baselines on the next step
anticipation. Finally, we also show that our model can successfully transfer
from text to the video domain zero-shot, ie, without fine-tuning or adaptation,
and produces good-quality future step predictions from video.

未来步骤预测问题的核心挑战是如何考虑自然场景中多种可能性的未来实现，该论文通过建立一个生成模型来拟合可能的下一步候选结果的分布，从而解决此问题。

GePSAn：烹饪视频中的生成式步骤预测

GePSAn: Generative Procedure Step Anticipation in Cooking Videos

We address the problem of extracting key steps from unlabeled procedural
videos, motivated by the potential of Augmented Reality (AR) headsets to
revolutionize job training and performance. We decompose the problem into two
steps: representation learning and key steps extraction. We propose a training
objective, Bootstrapped Multi-Cue Contrastive (BMC2) loss to learn
disciriminative representations for various steps without any labels. Different
from prior works, we develop techniques to train a light-weight temporal module
which uses off-the-shelf features for self supervision. Our approach can
seamlessly leverage information from multiple cues like optical flow, depth or
gaze to learn discriminative features for key-steps making it amenable for AR
applications. We finally extract key steps via a tunable algorithm that
clusters the representations and samples. We show significant improvements over
prior works for the task of key step localization and phase classification.
Qualitative results demonstrate that the extracted key steps are meaningful to
succinctly represent various steps of the procedural tasks.

通过提出的 Bootstrapped Multi-Cue Contrastive (BMC2) Loss，我们成功地建立了一个轻量级的时间模型，该模型使用现成的特征进行自我监督，并能够从多个提示信号中获取信息，最终实现从无标记程序性视频中提取出有意义的关键步骤。