The unprecedented surge in video data production in recent years necessitates
efficient tools to extract meaningful frames from videos for downstream tasks.
Long-term temporal reasoning is a key desideratum for frame retrieval systems.
While state-of-the-art foundation models, like VideoLLaMA and ViCLIP, are
proficient in short-term semantic understanding, they surprisingly fail at
long-term reasoning across frames. A key reason for this failure is that they
intertwine per-frame perception and temporal reasoning into a single deep
network. Hence, decoupling but co-designing semantic understanding and temporal
reasoning is essential for efficient scene identification. We propose a system
that leverages vision-language models for semantic understanding of individual
frames but effectively reasons about the long-term evolution of events using
state machines and temporal logic (TL) formulae that inherently capture memory.
Our TL-based reasoning improves the F1 score of complex event identification by
9-15% compared to benchmarks that use GPT4 for reasoning on state-of-the-art
self-driving datasets such as Waymo and NuScenes.

使用视觉语言模型进行语义理解，通过状态机和时间逻辑进行长期演变的事件推理，提高了复杂事件识别的 F1 得分。