This paper explores the challenge of procedure planning in instructional
videos, which involves creating goal-directed plans based on visual start and
goal observations from videos. Previous research has tackled this problem with
gradually weaker training supervision, from heavy intermediate visual
observations or language instructions to task class supervision. However, with
the advent of large language models, even given only the task name, these
models can produce a detailed plan. In this study, we propose a much weaker
setting without task name as supervision, which is not currently solvable by
existing large language models since they require good prompts with sufficient
information. Specifically, we hypothesize that previous intermediate
supervisions can serve as context information, and we use captions of visual
start and goal observations as a much cheaper form of supervision. This
approach greatly reduces the labeling cost since the captions can be easily
obtained by large pre-trained vision-language foundation models. Technically,
we apply BLIP to generate captions as supervision to train the context feature
with contrastive learning loss. Afterward, the context feature is fed into the
generator to aid in plan generation. Our experiments on two datasets with
varying scales demonstrate that our model can achieve comparable performance on
multiple metrics, which validates our hypothesis.

本研究探讨了在教学视频中的过程规划挑战，旨在通过观察视频中的视觉起点和目标，创建目标导向的计划。在先前的研究中，通过从大量的中间视觉观察或语言指令逐渐减少训练监督来解决此问题。然而，随着大型语言模型的出现，即使只提供任务名称，这些模型也可以生成详细的计划。本研究提出了一种更弱的设置方式，没有任务名称作为监督，当前大型语言模型无法解决此问题，因为它们需要具备充分信息的良好提示。具体而言，我们假设以前的中间监督可以作为上下文信息，并使用视觉起点和目标观察的标题作为更便宜的监督形式。该方法极大地降低了标记成本，因为标题可通过大型预训练视觉 - 语言模型轻松获得。从技术上讲，我们应用 BLIP 生成标题，作为对比学习损失训练上下文特征。然后，将上下文特征输入生成器以辅助计划生成。我们在两个不同规模的数据集上进行的实验证明，我们的模型在多个指标上可以达到可比较的性能，从而验证了我们的假设。