We introduce the video detours problem for navigating instructional videos.
Given a source video and a natural language query asking to alter the how-to
video's current path of execution in a certain way, the goal is to find a
related ''detour video'' that satisfies the requested alteration. To address
this challenge, we propose VidDetours, a novel video-language approach that
learns to retrieve the targeted temporal segments from a large repository of
how-to's using video-and-text conditioned queries. Furthermore, we devise a
language-based pipeline that exploits how-to video narration text to create
weakly supervised training data. We demonstrate our idea applied to the domain
of how-to cooking videos, where a user can detour from their current recipe to
find steps with alternate ingredients, tools, and techniques. Validating on a
ground truth annotated dataset of 16K samples, we show our model's significant
improvements over best available methods for video retrieval and question
answering, with recall rates exceeding the state of the art by 35%.

本研究提出了解决「视频绕道」问题的方法，通过学习视频和文本查询，检索出与之相关的视频片段，并利用基于文本的流程创建弱监督训练数据，进而在如何烹饪视频领域中实现了显著的检索和问题回答的性能提升。

导航教学视频的绕过方法

Detours for Navigating Instructional Videos

Procedural activity understanding requires perceiving human actions in terms
of a broader task, where multiple keysteps are performed in sequence across a
long video to reach a final goal state -- such as the steps of a recipe or a
DIY fix-it task. Prior work largely treats keystep recognition in isolation of
this broader structure, or else rigidly confines keysteps to align with a
predefined sequential script. We propose discovering a task graph automatically
from how-to videos to represent probabilistically how people tend to execute
keysteps, and then leverage this graph to regularize keystep recognition in
novel videos. On multiple datasets of real-world instructional videos, we show
the impact: more reliable zero-shot keystep localization and improved video
representation learning, exceeding the state of the art.

我们提出从视频中自动发现任务图表示人们如何执行关键步骤的概率，并利用该图来规范化新视频中的关键步骤识别，从而在多个真实世界教学视频数据集上展示了影响：更可靠的零样本关键步骤定位和改进的视频表示学习，超越了现有技术水平。