Automatically generating scripts (i.e. sequences of key steps described in
text) from video demonstrations and reasoning about the subsequent steps are
crucial to the modern AI virtual assistants to guide humans to complete
everyday tasks, especially unfamiliar ones. However, current methods for
generative script learning rely heavily on well-structured preceding steps
described in text and/or images or are limited to a certain domain, resulting
in a disparity with real-world user scenarios. To address these limitations, we
present a new benchmark challenge -- MultiScript, with two new tasks on
task-oriented multimodal script learning: (1) multimodal script generation, and
(2) subsequent step prediction. For both tasks, the input consists of a target
task name and a video illustrating what has been done to complete the target
task, and the expected output is (1) a sequence of structured step descriptions
in text based on the demonstration video, and (2) a single text description for
the subsequent step, respectively. Built from WikiHow, MultiScript covers
multimodal scripts in videos and text descriptions for over 6,655 human
everyday tasks across 19 diverse domains. To establish baseline performance on
MultiScript, we propose two knowledge-guided multimodal generative frameworks
that incorporate the task-related knowledge prompted from large language models
such as Vicuna. Experimental results show that our proposed approaches
significantly improve over the competitive baselines.

通过建立 MultiScript 基准测试，我们提出了两种引入任务相关知识的多模态生成框架，这些框架显著提升了脚本生成和后续步骤预测的性能。