Procedural activities are sequences of key-steps aimed at achieving specific
goals. They are crucial to build intelligent agents able to assist users
effectively. In this context, task graphs have emerged as a
human-understandable representation of procedural activities, encoding a
partial ordering over the key-steps. While previous works generally relied on
hand-crafted procedures to extract task graphs from videos, in this paper, we
propose an approach based on direct maximum likelihood optimization of edges'
weights, which allows gradient-based learning of task graphs and can be
naturally plugged into neural network architectures. Experiments on the
CaptainCook4D dataset demonstrate the ability of our approach to predict
accurate task graphs from the observation of action sequences, with an
improvement of +16.7% over previous approaches. Owing to the differentiability
of the proposed framework, we also introduce a feature-based approach, aiming
to predict task graphs from key-step textual or video embeddings, for which we
observe emerging video understanding abilities. Task graphs learned with our
approach are also shown to significantly enhance online mistake detection in
procedural egocentric videos, achieving notable gains of +19.8% and +7.5% on
the Assembly101 and EPIC-Tent datasets. Code for replicating experiments is
available at this https URL

通过最大似然优化边权重的方法，从动作序列中预测任务图，与之前的方法相比提高了 16.7% 的准确性；同时可通过文本或视频嵌入来预测任务图，并在程序性自我中心视频中显著提高在线错误检测的准确性，分别在 Assembly101 和 EPIC-Tent 数据集上获得了 19.8% 和 7.5% 的显著增益。

可微分任务图学习：程序化活动表示和基于自我感知视频的在线错误检测

Differentiable Task Graph Learning: Procedural Activity Representation  and Online Mistake Detection from Egocentric Videos

Following step-by-step procedures is an essential component of various
activities carried out by individuals in their daily lives. These procedures
serve as a guiding framework that helps to achieve goals efficiently, whether
it is assembling furniture or preparing a recipe. However, the complexity and
duration of procedural activities inherently increase the likelihood of making
errors. Understanding such procedural activities from a sequence of frames is a
challenging task that demands an accurate interpretation of visual information
and the ability to reason about the structure of the activity. To this end, we
collect a new egocentric 4D dataset, CaptainCook4D, comprising 384 recordings
(94.5 hours) of people performing recipes in real kitchen environments. This
dataset consists of two distinct types of activity: one in which participants
adhere to the provided recipe instructions and another in which they deviate
and induce errors. We provide 5.3K step annotations and 10K fine-grained action
annotations and benchmark the dataset for the following tasks: supervised error
recognition, multistep localization, and procedure learning

我们收集了一个全新的心理 4D 数据集 CaptainCook4D，包含了 384 个人在真实厨房环境中表演食谱的记录（共 94.5 小时），该数据集包含两种不同类型的活动：一种是参与者按照提供的食谱指示进行，另一种是参与者偏离指示并引发错误，我们提供了 5.3K 个步骤注释和 10K 个精细动作注释，并对该数据集进行了监督错误识别、多步本地化和程序学习的基准测试。

CaptainCook4D：过程活动错误理解数据集

CaptainCook4D: A dataset for understanding errors in procedural  activities

In this paper we present an approach for localizing steps of procedural
activities in narrated how-to videos. To deal with the scarcity of labeled data
at scale, we source the step descriptions from a language knowledge base
(wikiHow) containing instructional articles for a large variety of procedural
tasks. Without any form of manual supervision, our model learns to temporally
ground the steps of procedural articles in how-to videos by matching three
modalities: frames, narrations, and step descriptions. Specifically, our method
aligns steps to video by fusing information from two distinct pathways: i) {\em
direct} alignment of step descriptions to frames, ii) {\em indirect} alignment
obtained by composing steps-to-narrations with narrations-to-video
correspondences. Notably, our approach performs global temporal grounding of
all steps in an article at once by exploiting order information, and is trained
with step pseudo-labels which are iteratively refined and aggressively
filtered. In order to validate our model we introduce a new evaluation
benchmark -- HT-Step -- obtained by manually annotating a 124-hour subset of
HowTo100M\footnote{A test server is accessible at
https://eval.ai/web/challenges/challenge-page/2082.} with steps sourced
from wikiHow articles. Experiments on this benchmark as well as zero-shot
evaluations on CrossTask demonstrate that our multi-modality alignment yields
dramatic gains over several baselines and prior works. Finally, we show that
our inner module for matching narration-to-video outperforms by a large margin
the state of the art on the HTM-Align narration-video alignment benchmark.

本文提出了一种无需手动监督的方法来将『如何做』视频中的步骤与 wikiHow 等语言知识库中的指导文章进行对齐，并采用多模态对齐进行全局时间定位，验证结果表明与现有方法相比具有明显优势。