Following step-by-step procedures is an essential component of various
activities carried out by individuals in their daily lives. These procedures
serve as a guiding framework that helps to achieve goals efficiently, whether
it is assembling furniture or preparing a recipe. However, the complexity and
duration of procedural activities inherently increase the likelihood of making
errors. Understanding such procedural activities from a sequence of frames is a
challenging task that demands an accurate interpretation of visual information
and the ability to reason about the structure of the activity. To this end, we
collect a new egocentric 4D dataset, CaptainCook4D, comprising 384 recordings
(94.5 hours) of people performing recipes in real kitchen environments. This
dataset consists of two distinct types of activity: one in which participants
adhere to the provided recipe instructions and another in which they deviate
and induce errors. We provide 5.3K step annotations and 10K fine-grained action
annotations and benchmark the dataset for the following tasks: supervised error
recognition, multistep localization, and procedure learning

我们收集了一个全新的心理 4D 数据集 CaptainCook4D，包含了 384 个人在真实厨房环境中表演食谱的记录（共 94.5 小时），该数据集包含两种不同类型的活动：一种是参与者按照提供的食谱指示进行，另一种是参与者偏离指示并引发错误，我们提供了 5.3K 个步骤注释和 10K 个精细动作注释，并对该数据集进行了监督错误识别、多步本地化和程序学习的基准测试。