Procedural activities are sequences of key-steps aimed at achieving specific
goals. They are crucial to build intelligent agents able to assist users
effectively. In this context, task graphs have emerged as a
human-understandable representation of procedural activities, encoding a
partial ordering over the key-steps. While previous works generally relied on
hand-crafted procedures to extract task graphs from videos, in this paper, we
propose an approach based on direct maximum likelihood optimization of edges'
weights, which allows gradient-based learning of task graphs and can be
naturally plugged into neural network architectures. Experiments on the
CaptainCook4D dataset demonstrate the ability of our approach to predict
accurate task graphs from the observation of action sequences, with an
improvement of +16.7% over previous approaches. Owing to the differentiability
of the proposed framework, we also introduce a feature-based approach, aiming
to predict task graphs from key-step textual or video embeddings, for which we
observe emerging video understanding abilities. Task graphs learned with our
approach are also shown to significantly enhance online mistake detection in
procedural egocentric videos, achieving notable gains of +19.8% and +7.5% on
the Assembly101 and EPIC-Tent datasets. Code for replicating experiments is
available at this https URL

通过最大似然优化边权重的方法，从动作序列中预测任务图，与之前的方法相比提高了 16.7% 的准确性；同时可通过文本或视频嵌入来预测任务图，并在程序性自我中心视频中显著提高在线错误检测的准确性，分别在 Assembly101 和 EPIC-Tent 数据集上获得了 19.8% 和 7.5% 的显著增益。

可微分任务图学习：程序化活动表示和基于自我感知视频的在线错误检测

Differentiable Task Graph Learning: Procedural Activity Representation  and Online Mistake Detection from Egocentric Videos

This work explores the problem of generating task graphs of real-world
activities. Different from prior formulations, we consider a setting where text
transcripts of instructional videos performing a real-world activity (e.g.,
making coffee) are provided and the goal is to identify the key steps relevant
to the task as well as the dependency relationship between these key steps. We
propose a novel task graph generation approach that combines the reasoning
capabilities of instruction-tuned language models along with clustering and
ranking components to generate accurate task graphs in a completely
unsupervised manner. We show that the proposed approach generates more accurate
task graphs compared to a supervised learning approach on tasks from the ProceL
and CrossTask datasets.

本文探讨了生成真实世界活动任务图的问题，提出了一种无监督的任务图生成方法，结合语言模型的推理能力、聚类和排名组件，比 ProceL 和 CrossTask 数据集中的有监督学习方法生成更准确的任务图。

从教学视频转录中生成无监督任务图

Unsupervised Task Graph Generation from Instructional Video Transcripts

Machine learning applications are increasingly deployed not only to serve
predictions using static models, but also as tightly-integrated components of
feedback loops involving dynamic, real-time decision making. These applications
pose a new set of requirements, none of which are difficult to achieve in
isolation, but the combination of which creates a challenge for existing
distributed execution frameworks: computation with millisecond latency at high
throughput, adaptive construction of arbitrary task graphs, and execution of
heterogeneous kernels over diverse sets of resources. We assert that a new
distributed execution framework is needed for such ML applications and propose
a candidate approach with a proof-of-concept architecture that achieves a 63x
performance improvement over a state-of-the-art execution framework for a
representative application.

针对机器学习在实时决策中的应用，提出需要一种新的分布式执行框架来实现毫秒延迟、高吞吐、任务图的自适应构建以及异构内核的执行等一系列要求，并提出了一个候选方案，其架构已经获得了 63 倍的性能提升。