Multimodal Large Language Models (MLLMs) have shown significant promise in
various applications, leading to broad interest from researchers and
practitioners alike. However, a comprehensive evaluation of their long-context
capabilities remains underexplored. To address these gaps, we introduce the
MultiModal Needle-in-a-haystack (MMNeedle) benchmark, specifically designed to
assess the long-context capabilities of MLLMs. Besides multi-image input, we
employ image stitching to further increase the input context length, and
develop a protocol to automatically generate labels for sub-image level
retrieval. Essentially, MMNeedle evaluates MLLMs by stress-testing their
capability to locate a target sub-image (needle) within a set of images
(haystack) based on textual instructions and descriptions of image contents.
This setup necessitates an advanced understanding of extensive visual contexts
and effective information retrieval within long-context image inputs. With this
benchmark, we evaluate state-of-the-art MLLMs, encompassing both API-based and
open-source models. The findings reveal that GPT-4o consistently surpasses
other models in long-context scenarios, but suffers from hallucination problems
in negative samples, i.e., when needles are not in the haystacks. Our
comprehensive long-context evaluation of MLLMs also sheds lights on the
considerable performance gap between API-based and open-source models. All the
code, data, and instructions required to reproduce the main results are
available at this https URL

对多模态大语言模型进行了全面评估，介绍了一种用于长上下文能力评估的基准测试方法，包括多图像输入和图像拼接，以加大输入上下文长度，并开发了一个自动生成子图像级标签的协议。通过基于文本指令和图像内容描述，对 MLLMs 在长上下文图像输入中在一组图像中定位目标子图像的能力进行了压力测试。结果显示 GPT-4o 在长上下文情景中始终优于其他模型，但在负样本（即针未在抽屉中）中存在错误信息生成问题。此外，对 API 模型和开源模型进行了综合的长上下文评估，显示了相当大的性能差距。可通过提供的网址获取代码、数据和重现主要结果的指令。