In this paper we consider the problem of classifying fine-grained, multi-step
activities (e.g., cooking different recipes, making disparate home
improvements, creating various forms of arts and crafts) from long videos
spanning up to several minutes. Accurately categorizing these activities
requires not only recognizing the individual steps that compose the task but
also capturing their temporal dependencies. This problem is dramatically
different from traditional action classification, where models are typically
optimized on videos that span only a few seconds and that are manually trimmed
to contain simple atomic actions. While step annotations could enable the
training of models to recognize the individual steps of procedural activities,
existing large-scale datasets in this area do not include such segment labels
due to the prohibitive cost of manually annotating temporal boundaries in long
videos. To address this issue, we propose to automatically identify steps in
instructional videos by leveraging the distant supervision of a textual
knowledge base (wikiHow) that includes detailed descriptions of the steps
needed for the execution of a wide variety of complex activities. Our method
uses a language model to match noisy, automatically-transcribed speech from the
video to step descriptions in the knowledge base. We demonstrate that video
models trained to recognize these automatically-labeled steps (without manual
supervision) yield a representation that achieves superior generalization
performance on four downstream tasks: recognition of procedural activities,
step classification, step forecasting and egocentric video classification.

本文研究了从长达数分钟的视频中识别精细、多步骤活动的问题，通过远程监督的语言模型方法，基于 wikiHow 的文本数据库自动标注视频中的步骤，并在识别过程中考虑了它们的时间依赖性，实现了较高的泛化性能。