Automatically generating scripts (i.e. sequences of key steps described in
text) from video demonstrations and reasoning about the subsequent steps are
crucial to the modern AI virtual assistants to guide humans to complete
everyday tasks, especially unfamiliar ones. However, current methods for
generative script learning rely heavily on well-structured preceding steps
described in text and/or images or are limited to a certain domain, resulting
in a disparity with real-world user scenarios. To address these limitations, we
present a new benchmark challenge -- MultiScript, with two new tasks on
task-oriented multimodal script learning: (1) multimodal script generation, and
(2) subsequent step prediction. For both tasks, the input consists of a target
task name and a video illustrating what has been done to complete the target
task, and the expected output is (1) a sequence of structured step descriptions
in text based on the demonstration video, and (2) a single text description for
the subsequent step, respectively. Built from WikiHow, MultiScript covers
multimodal scripts in videos and text descriptions for over 6,655 human
everyday tasks across 19 diverse domains. To establish baseline performance on
MultiScript, we propose two knowledge-guided multimodal generative frameworks
that incorporate the task-related knowledge prompted from large language models
such as Vicuna. Experimental results show that our proposed approaches
significantly improve over the competitive baselines.

通过建立 MultiScript 基准测试，我们提出了两种引入任务相关知识的多模态生成框架，这些框架显著提升了脚本生成和后续步骤预测的性能。

MULTISCRIPT: 支持开放领域日常任务的多模态脚本学习

MULTISCRIPT: Multimodal Script Learning for Supporting Open Domain  Everyday Tasks

Goal-oriented generative script learning aims to generate subsequent steps
based on a goal, which is an essential task to assist robots in performing
stereotypical activities of daily life. We show that the performance of this
task can be improved if historical states are not just captured by the
linguistic instructions given to people, but are augmented with the additional
information provided by accompanying images. Therefore, we propose a new task,
Multimedia Generative Script Learning, to generate subsequent steps by tracking
historical states in both text and vision modalities, as well as presenting the
first benchmark containing 2,338 tasks and 31,496 steps with descriptive
images. We aim to generate scripts that are visual-state trackable, inductive
for unseen tasks, and diverse in their individual steps. We propose to encode
visual state changes through a multimedia selective encoder, transferring
knowledge from previously observed tasks using a retrieval-augmented decoder,
and presenting the distinct information at each step by optimizing a
diversity-oriented contrastive learning objective. We define metrics to
evaluate both generation quality and inductive quality. Experiment results
demonstrate that our approach significantly outperforms strong baselines.

提出了一种新的多媒体生成式脚本学习任务，通过同时跟踪文本和视觉模态中的历史状态，并以描述性图像呈现，旨在生成可视状态跟踪、对未见任务有归纳性和具有多样性的脚本，通过多媒体选性编码器实现对视觉状态变化的编码，使用检索增强解码器从以前观察到的任务中传递知识，通过优化多样性导向的对比学习目标，呈现每个步骤的不同信息。实验结果表明，该方法显著优于强基线。