Comparing a user video to a reference how-to video is a key requirement for
AR/VR technology delivering personalized assistance tailored to the user's
progress. However, current approaches for language-based assistance can only
answer questions about a single video. We propose an approach that first
automatically generates large amounts of visual instruction tuning data
involving pairs of videos from HowTo100M by leveraging existing step
annotations and accompanying narrations, and then trains a video-conditioned
language model to jointly reason across multiple raw videos. Our model achieves
state-of-the-art performance at identifying differences between video pairs and
ranking videos based on the severity of these differences, and shows promising
ability to perform general reasoning over multiple videos.

通过利用现有步骤注释和配套叙述，我们提出了一种方法，首先自动生成大量涉及来自 HowTo100M 的视频对的视觉指导调优数据，然后训练一个视频修饰的语言模型以联合推理多个原始视频，以识别视频对之间的差异并根据这些差异的严重程度对视频进行排名，显示出在多个视频上进行一般推理的有希望能力。

教学视频中的步骤差异

Step Differences in Instructional Video

Driver reaction is of vital importance in risk scenarios. Drivers can take
correct evasive maneuver at proper cushion time to avoid the potential traffic
crashes, but this reaction process is highly experience-dependent and requires
various levels of driving skills. To improve driving safety and avoid the
traffic accidents, it is necessary to provide all road drivers with on-board
driving assistance. This study explores the plausibility of case-based
reasoning (CBR) as the inference paradigm underlying the choice of personalized
crash evasive maneuvers and the cushion time, by leveraging the wealthy of
human driving experience from the steady stream of traffic cases, which have
been rarely explored in previous studies. To this end, in this paper, we
propose an open evolving framework for generating personalized on-board driving
assistance. In particular, we present the FFMTE model with high performance to
model the traffic events and build the case database; A tailored CBR-based
method is then proposed to retrieve, reuse and revise the existing cases to
generate the assistance. We take the 100-Car Naturalistic Driving Study dataset
as an example to build and test our framework; the experiments show reasonable
results, providing the drivers with valuable evasive information to avoid the
potential crashes in different scenarios.

该研究探索了基于案例推理（CBR）作为推断范式的合理性，以选择个性化的撞车回避动作和缓冲时间，旨在提供个性化的驾驶辅助，避免交通事故。提出了一个高效的 FFMTE 模型来建模交通事件和构建案例数据库，并提出了一种特定的 CBR 方法来检索、重复使用和修订现有案例以生成辅助信息。在 100 辆汽车自然行驶研究数据集的例子中建立和测试了该框架；实验结果表明，在不同情境下为司机提供有价值的避难信息。