A key challenge with procedure planning in instructional videos lies in how
to handle a large decision space consisting of a multitude of action types that
belong to various tasks. To understand real-world video content, an AI agent
must proficiently discern these action types (e.g., pour milk, pour water, open
lid, close lid, etc.) based on brief visual observation. Moreover, it must
adeptly capture the intricate semantic relation of the action types and task
goals, along with the variable action sequences. Recently, notable progress has
been made via the integration of diffusion models and visual representation
learning to address the challenge. However, existing models employ rudimentary
mechanisms to utilize task information to manage the decision space. To
overcome this limitation, we introduce a simple yet effective enhancement - a
masked diffusion model. The introduced mask acts akin to a task-oriented
attention filter, enabling the diffusion/denoising process to concentrate on a
subset of action types. Furthermore, to bolster the accuracy of task
classification, we harness more potent visual representation learning
techniques. In particular, we learn a joint visual-text embedding, where a text
embedding is generated by prompting a pre-trained vision-language model to
focus on human actions. We evaluate the method on three public datasets and
achieve state-of-the-art performance on multiple metrics. Code is available at
this https URL

在教学视频的程序规划中，处理由多个属于不同任务的动作类型组成的决策空间是一个关键挑战。为了理解现实世界的视频内容，人工智能代理系统必须能够根据简要的视觉观察熟练地区分这些动作类型，并能够捕捉这些动作类型与任务目标之间的复杂语义关系以及可变的动作序列。最近，通过集成扩散模型和视觉表示学习取得了显著进展来应对这一挑战。然而，现有模型使用了基本机制来利用任务信息来管理决策空间。为了克服这个限制，我们引入了一种简单而有效的增强方法 - 掩蔽扩散模型。引入的掩蔽机制类似于一种面向任务的注意力过滤器，使得扩散 / 去噪过程能够集中在一部分动作类型上。此外，为了增强任务分类的准确性，我们采用了更强大的视觉表示学习技术。特别地，我们学习了一个联合视觉 - 文本嵌入，其中文本嵌入是通过提示一个预先训练的视觉 - 语言模型关注人类动作来生成的。我们在三个公共数据集上评估了该方法，并在多个指标上取得了最先进的性能。