Following step-by-step procedures is an essential component of various
activities carried out by individuals in their daily lives. These procedures
serve as a guiding framework that helps to achieve goals efficiently, whether
it is assembling furniture or preparing a recipe. However, the complexity and
duration of procedural activities inherently increase the likelihood of making
errors. Understanding such procedural activities from a sequence of frames is a
challenging task that demands an accurate interpretation of visual information
and the ability to reason about the structure of the activity. To this end, we
collect a new egocentric 4D dataset, CaptainCook4D, comprising 384 recordings
(94.5 hours) of people performing recipes in real kitchen environments. This
dataset consists of two distinct types of activity: one in which participants
adhere to the provided recipe instructions and another in which they deviate
and induce errors. We provide 5.3K step annotations and 10K fine-grained action
annotations and benchmark the dataset for the following tasks: supervised error
recognition, multistep localization, and procedure learning

我们收集了一个全新的心理 4D 数据集 CaptainCook4D，包含了 384 个人在真实厨房环境中表演食谱的记录（共 94.5 小时），该数据集包含两种不同类型的活动：一种是参与者按照提供的食谱指示进行，另一种是参与者偏离指示并引发错误，我们提供了 5.3K 个步骤注释和 10K 个精细动作注释，并对该数据集进行了监督错误识别、多步本地化和程序学习的基准测试。

CaptainCook4D：过程活动错误理解数据集

CaptainCook4D: A dataset for understanding errors in procedural  activities

Given multiple videos of the same task, procedure learning addresses
identifying the key-steps and determining their order to perform the task. For
this purpose, existing approaches use the signal generated from a pair of
videos. This makes key-steps discovery challenging as the algorithms lack
inter-videos perspective. Instead, we propose an unsupervised Graph-based
Procedure Learning (GPL) framework. GPL consists of the novel UnityGraph that
represents all the videos of a task as a graph to obtain both intra-video and
inter-videos context. Further, to obtain similar embeddings for the same
key-steps, the embeddings of UnityGraph are updated in an unsupervised manner
using the Node2Vec algorithm. Finally, to identify the key-steps, we cluster
the embeddings using KMeans. We test GPL on benchmark ProceL, CrossTask, and
EgoProceL datasets and achieve an average improvement of 2% on third-person
datasets and 3.6% on EgoProceL over the state-of-the-art.

给定相同任务的多个视频，本研究致力于确定任务的关键步骤并确定其顺序。为此，我们提出了一种无监督的基于图的过程学习 (GPL) 框架，其中包括了新颖的 UnityGraph 来获取视频内和视频间上下文。另外，为了获得相似的嵌入，我们使用 Node2Vec 算法无监督地更新 UnityGraph 的嵌入向量。最后，我们使用 KMeans 算法对嵌入向量进行聚类以确定关键步骤。我们在 ProceL、CrossTask 和 EgoProceL 数据集上进行了测试，并相比最先进方法，第三方数据集的平均改进为 2%，EgoProceL 数据集的改进为 3.6%。

团结则存，分裂则亡：UnityGraph 用于无监督视频程序学习

United We Stand, Divided We Fall: UnityGraph for Unsupervised Procedure  Learning from Videos

Procedure learning involves identifying the key-steps and determining their
logical order to perform a task. Existing approaches commonly use third-person
videos for learning the procedure, making the manipulated object small in
appearance and often occluded by the actor, leading to significant errors. In
contrast, we observe that videos obtained from first-person (egocentric)
wearable cameras provide an unobstructed and clear view of the action. However,
procedure learning from egocentric videos is challenging because (a) the camera
view undergoes extreme changes due to the wearer's head motion, and (b) the
presence of unrelated frames due to the unconstrained nature of the videos. Due
to this, current state-of-the-art methods' assumptions that the actions occur
at approximately the same time and are of the same duration, do not hold.
Instead, we propose to use the signal provided by the temporal correspondences
between key-steps across videos. To this end, we present a novel
self-supervised Correspond and Cut (CnC) framework for procedure learning. CnC
identifies and utilizes the temporal correspondences between the key-steps
across multiple videos to learn the procedure. Our experiments show that CnC
outperforms the state-of-the-art on the benchmark ProceL and CrossTask datasets
by 5.2% and 6.3%, respectively. Furthermore, for procedure learning using
egocentric videos, we propose the EgoProceL dataset consisting of 62 hours of
videos captured by 130 subjects performing 16 tasks. The source code and the
dataset are available on the project page this https URL.

本文提出了一种使用基于第一人称视角拍摄的视频学习过程的框架，该框架可以在多个视频之间识别和利用关键步骤之间的时间对应关系，从而对过程进行学习。通过对两个基准数据集的实验，证明这一框架的性能优于现有的同类方法。此外，本文还提出了一个用于使用第一人称视角拍摄的视频进行过程学习的数据集。