Recently, with the rise of web videos, managing and understanding large-scale
video datasets has become increasingly important. Video Large Language Models
(VideoLLMs) have emerged in recent years due to their strong video
understanding capabilities. However, training and inference processes for
VideoLLMs demand vast amounts of data, presenting significant challenges to
data management, particularly regarding efficiency, robustness, and
effectiveness. In this work, we present KeyVideoLLM, a text-video frame
similarity-based keyframe selection method designed to manage VideoLLM data
efficiently, robustly, and effectively. Specifically, KeyVideoLLM achieves a
remarkable data compression rate of up to 60.9 times, substantially lowering
disk space requirements, which proves its high efficiency. Additionally, it
maintains a 100% selection success rate across all video formats and scales,
enhances processing speed by up to 200 times compared to existing keyframe
selection methods, and does not require hyperparameter tuning. Beyond its
outstanding efficiency and robustness, KeyVideoLLM further improves model
performance in video question-answering tasks during both training and
inference stages. Notably, it consistently achieved the state-of-the-art (SoTA)
experimental results on diverse datasets.

通过基于文本 - 视频帧相似度的关键帧选择方法，KeyVideoLLM 能够高效、鲁棒、有效地管理 VideoLLM 数据，并在视频问答任务中显著改善模型性能，实验结果一直保持在最新技术的前沿。

KeyVideoLLM：面向大规模视频关键帧选择

KeyVideoLLM: Towards Large-scale Video Keyframe Selection

Recently, Multimodal Large Language Models (MLLMs) have been used as agents
to control keyboard and mouse inputs by directly perceiving the Graphical User
Interface (GUI) and generating corresponding code. However, current agents
primarily exhibit excellent understanding capabilities in static environments
and are predominantly applied in relatively simple domains, such as Web or
mobile interfaces. We argue that a robust GUI agent should be capable of
perceiving temporal information on the GUI, including dynamic Web content and
multi-step tasks. Additionally, it should possess a comprehensive understanding
of various GUI scenarios, including desktop software and multi-window
interactions. To this end, this paper introduces a new dataset, termed
GUI-World, which features meticulously crafted Human-MLLM annotations,
extensively covering six GUI scenarios and eight types of GUI-oriented
questions in three formats. We evaluate the capabilities of current
state-of-the-art MLLMs, including ImageLLMs and VideoLLMs, in understanding
various types of GUI content, especially dynamic and sequential content. Our
findings reveal that ImageLLMs struggle with dynamic GUI content without
manually annotated keyframes or operation history. On the other hand, VideoLLMs
fall short in all GUI-oriented tasks given the sparse GUI video dataset. Based
on GUI-World, we take the initial step of leveraging a fine-tuned VideoLLM as a
GUI agent, demonstrating an improved understanding of various GUI tasks.
However, due to the limitations in the performance of base LLMs, we conclude
that using VideoLLMs as GUI agents remains a significant challenge. We believe
our work provides valuable insights for future research in dynamic GUI content
understanding. The code and dataset are publicly available at our project
homepage: this https URL

该研究论文介绍了一个新的数据集，名为 GUI-World，包含六种 GUI 场景和八种 GUI 问题类型的人工 - MLLM 注释，评估了当前最先进的 MLLMs（包括 ImageLLMs 和 VideoLLMs）在理解各种 GUI 内容方面的能力，发现 ImageLLMs 在没有手动注释的关键帧或操作历史的情况下难以处理动态 GUI 内容，而 VideoLLMs 在 GUI 视频数据集稀缺的情况下在所有 GUI 任务中表现不佳。然而，作者通过利用 Fine-tuned VideoLLM 作为 GUI agent 迈出了一步，展示了对各种 GUI 任务的改进理解，但由于基础 LLMs 的性能限制，使用 VideoLLMs 作为 GUI agent 仍然是一个重大挑战，研究为未来动态 GUI 内容理解的研究提供了有价值的见解。

GUI-WORLD: GUI 导向的多模态基于 LLM 的 Agent 数据集

GUI-WORLD: A Dataset for GUI-oriented Multimodal LLM-based Agents

Empowered by Large Language Models (LLMs), recent advancements in VideoLLMs
have driven progress in various video understanding tasks. These models encode
video representations through pooling or query aggregation over a vast number
of visual tokens, making computational and memory costs affordable. Despite
successfully providing an overall comprehension of video content, existing
VideoLLMs still face challenges in achieving detailed understanding in videos
due to overlooking local information in long-term videos. To tackle this
challenge, we introduce LongVLM, a straightforward yet powerful VideoLLM for
long video understanding, building upon the observation that long videos often
consist of sequential key events, complex actions, and camera movements. Our
approach proposes to decompose long videos into multiple short-term segments
and encode local features for each local segment via a hierarchical token
merging module. These features are concatenated in temporal order to maintain
the storyline across sequential short-term segments. Additionally, we propose
to integrate global semantics into each local feature to enhance context
understanding. In this way, we encode video representations that incorporate
both local and global information, enabling the LLM to generate comprehensive
responses for long-term videos. Experimental results on the VideoChatGPT
benchmark and zero-shot video question-answering datasets demonstrate the
superior capabilities of our model over the previous state-of-the-art methods.
Qualitative examples demonstrate that our model produces more precise responses
for long videos understanding. Code is available at
https://github.com/ziplab/LongVLM.

使用长视频理解任务中的 Large Language Models（LLMs）面临的挑战，本文提出了一种名为 LongVLM 的 VideoLLM 模型，通过分解长视频为短期片段，并使用分层令牌合并模块编码局部特征，维护顺序，整合全局语义信息，实现对长期视频的全面理解。实验证明了该模型在视频理解任务中的优越性能。