In this paper, we explore the capability of an agent to construct a logical
sequence of action steps, thereby assembling a strategic procedural plan. This
plan is crucial for navigating from an initial visual observation to a target
visual outcome, as depicted in real-life instructional videos. Existing works
have attained partial success by extensively leveraging various sources of
information available in the datasets, such as heavy intermediate visual
observations, procedural names, or natural language step-by-step instructions,
for features or supervision signals. However, the task remains formidable due
to the implicit causal constraints in the sequencing of steps and the
variability inherent in multiple feasible plans. To tackle these intricacies
that previous efforts have overlooked, we propose to enhance the capabilities
of the agent by infusing it with procedural knowledge. This knowledge, sourced
from training procedure plans and structured as a directed weighted graph,
equips the agent to better navigate the complexities of step sequencing and its
potential variations. We coin our approach KEPP, a novel Knowledge-Enhanced
Procedure Planning system, which harnesses a probabilistic procedural knowledge
graph extracted from training data, effectively acting as a comprehensive
textbook for the training domain. Experimental evaluations across three
widely-used datasets under settings of varying complexity reveal that KEPP
attains superior, state-of-the-art results while requiring only minimal
supervision.

该研究探讨了代理程序在构建逻辑动作步骤序列方面的能力，从而组装战略性的过程计划。利用来自训练数据的程序化知识图，我们提出了一种名为 KEPP 的新型知识增强程序规划系统，该系统在各种复杂度设置下的实验评估结果表明，KEPP 取得了卓越的、最新的结果，同时只需要很少的监督。

为何不使用教科书？知识增强的教学视频程序规划

Why Not Use Your Textbook? Knowledge-Enhanced Procedure Planning of  Instructional Videos

Training Large Language Models (LLMs) to follow user instructions has been
shown to supply the LLM with ample capacity to converse fluently while being
aligned with humans. Yet, it is not completely clear how an LLM can lead a
plan-grounded conversation in mixed-initiative settings where instructions flow
in both directions of the conversation, i.e. both the LLM and the user provide
instructions to one another. In this paper, we tackle a dual goal
mixed-initiative conversational setting where the LLM not only grounds the
conversation on an arbitrary plan but also seeks to satisfy both a procedural
plan and user instructions. The LLM is then responsible for guiding the user
through the plan and, at the same time, adapting to new circumstances,
answering questions, and activating safety guardrails when needed. We propose a
novel LLM that grounds the dialogue on a procedural plan, can take the dialogue
initiative, and enforces guardrails on the system's behavior, while also
improving the LLM's responses to unexpected user behavior. Experiments in
controlled settings and with real users show that the best-performing model,
which we call PlanLLM, achieves a 2.1x improvement over a strong baseline.
Moreover, experiments also show good generalization to unseen domains.

本文介绍了一个新颖的大型语言模型（LLM），在混合主动对话环境中对话，并通过制定规程计划引导用户，并在需要时激活安全保障措施。实验结果表明，该模型取得了 2.1 倍的改进，并在未知领域中显示出良好的泛化能力。