Recent progress in multimodal large language models has markedly enhanced the
understanding of short videos (typically under one minute), and several
evaluation datasets have emerged accordingly. However, these advancements fall
short of meeting the demands of real-world applications such as embodied
intelligence for long-term decision-making, in-depth movie reviews and
discussions, and live sports commentary, all of which require comprehension of
long videos spanning several hours. To address this gap, we introduce LVBench,
a benchmark specifically designed for long video understanding. Our dataset
comprises publicly sourced videos and encompasses a diverse set of tasks aimed
at long video comprehension and information extraction. LVBench is designed to
challenge multimodal models to demonstrate long-term memory and extended
comprehension capabilities. Our extensive evaluations reveal that current
multimodal models still underperform on these demanding long video
understanding tasks. Through LVBench, we aim to spur the development of more
advanced models capable of tackling the complexities of long video
comprehension. Our data and code are publicly available at:
this https URL

多模态大型语言模型的最新进展在短视频（通常不超过一分钟）的理解方面取得了显著的提升，并相应产生了多个评估数据集。然而，这些进展还不足以满足现实世界应用的需求，如长期决策的具身智能、深入的电影评论和讨论，以及现场体育评论，这些都需要对数小时的长视频进行理解。为了弥补这一差距，我们介绍了专门设计用于长视频理解的基准测试集 LVBench。我们的数据集包括来源于公共的视频，囊括了一组多样化的任务，旨在进行长视频理解和信息抽取。LVBench 旨在挑战多模态模型展示长期记忆和扩展理解能力。我们进行了大量的评估，结果显示当前的多模态模型在这些具有挑战性的长视频理解任务上表现不佳。通过 LVBench，我们旨在推动更先进的模型的发展，以应对长视频理解的复杂性。我们的数据和代码在以下链接公开可用：this https URL

LVBench：极长视频理解基准

LVBench: An Extreme Long Video Understanding Benchmark

The advent of AI driven large language models (LLMs) have stirred discussions
about their role in qualitative research. Some view these as tools to enrich
human understanding, while others perceive them as threats to the core values
of the discipline. This study aimed to compare and contrast the comprehension
capabilities of humans and LLMs. We conducted an experiment with small sample
of Alexa app reviews, initially classified by a human analyst. LLMs were then
asked to classify these reviews and provide the reasoning behind each
classification. We compared the results with human classification and
reasoning. The research indicated a significant alignment between human and
ChatGPT 3.5 classifications in one third of cases, and a slightly lower
alignment with GPT4 in over a quarter of cases. The two AI models showed a
higher alignment, observed in more than half of the instances. However, a
consensus across all three methods was seen only in about one fifth of the
classifications. In the comparison of human and LLMs reasoning, it appears that
human analysts lean heavily on their individual experiences. As expected, LLMs,
on the other hand, base their reasoning on the specific word choices found in
app reviews and the functional components of the app itself. Our results
highlight the potential for effective human LLM collaboration, suggesting a
synergistic rather than competitive relationship. Researchers must continuously
evaluate LLMs role in their work, thereby fostering a future where AI and
humans jointly enrich qualitative research.

本研究对语言模型（LLMs）的理解能力进行了比较和对照，发现人类分析师和 LLMs 的分类和推理能力存在显著差异，但二者合作可能会产生协同效应，从而丰富了定性研究。