Large Language Models (LLMs), known to contain a strong awareness of world
knowledge, have allowed recent approaches to achieve excellent performance on
Long-Video Understanding benchmarks, but at high inference costs. In this work,
we first propose Likelihood Selection, a simple technique that unlocks faster
inference in autoregressive LLMs for multiple-choice tasks common in long-video
benchmarks. In addition to faster inference, we discover the resulting models
to yield surprisingly good accuracy on long-video tasks, even with no video
specific information. Building on this, we inject video-specific object-centric
information extracted from off-the-shelf pre-trained models and utilize natural
language as a medium for information fusion. Our resulting Multimodal Video
Understanding (MVU) framework demonstrates state-of-the-art performance across
long-video and fine-grained action recognition benchmarks. Code available at:
this https URL

利用多模态视频理解框架，在长视频和细粒度动作识别基准测试中展示了最前沿的性能表现。

一个多模态语言模型中理解长视频的方法

Understanding Long Videos in One Multimodal Language Model Pass

Multimodal video understanding is crucial for analyzing egocentric videos,
where integrating multiple sensory signals significantly enhances action
recognition and moment localization. However, practical applications often
grapple with incomplete modalities due to factors like privacy concerns,
efficiency demands, or hardware malfunctions. Addressing this, our study delves
into the impact of missing modalities on egocentric action recognition,
particularly within transformer-based models. We introduce a novel concept
-Missing Modality Token (MMT)-to maintain performance even when modalities are
absent, a strategy that proves effective in the Ego4D, Epic-Kitchens, and
Epic-Sounds datasets. Our method mitigates the performance loss, reducing it
from its original $\sim 30\%$ drop to only $\sim 10\%$ when half of the test
set is modal-incomplete. Through extensive experimentation, we demonstrate the
adaptability of MMT to different training scenarios and its superiority in
handling missing modalities compared to current methods. Our research
contributes a comprehensive analysis and an innovative approach, opening
avenues for more resilient multimodal systems in real-world settings.

本研究通过引入 Missing Modality Token（MMT）的新概念，综合分析了缺失模态对自我中心动作识别的影响，并证明了 MMT 方法在处理缺失模态时的优越性。

探索多模式自为视角数据集中的缺失模态

Exploring Missing Modality in Multimodal Egocentric Datasets

Existing deep video models are limited by specific tasks, fixed input-output
spaces, and poor generalization capabilities, making it difficult to deploy
them in real-world scenarios. In this paper, we present our vision for
multimodal and versatile video understanding and propose a prototype system,
\system. Our system is built upon a tracklet-centric paradigm, which treats
tracklets as the basic video unit and employs various Video Foundation Models
(ViFMs) to annotate their properties e.g., appearance, motion, \etc. All the
detected tracklets are stored in a database and interact with the user through
a database manager. We have conducted extensive case studies on different types
of in-the-wild videos, which demonstrates the effectiveness of our method in
answering various video-related problems. Our project is available at
this https URL

本文提出了一种基于轨迹为中心的多模态视频理解原型系统，使用各种视频基础模型（ViFMs）注释其特性，存储在数据库中，并通过数据库管理器与用户交互，以解决各种视频相关问题。