Multimodal Large Language Models (MLLMs) have achieved SOTA performance in
various visual language tasks by fusing the visual representations with LLMs
leveraging some visual adapters. In this paper, we first establish that
adapters using query-based Transformers such as Q-former is a simplified
Multi-instance Learning method without considering instance
heterogeneity/correlation. We then propose a general component termed
Multi-instance Visual Prompt Generator (MIVPG) to incorporate enriched visual
representations into LLMs by taking advantage of instance correlation between
images or patches for the same sample. Quantatitive evaluation on three public
vision-language (VL) datasets from different scenarios shows that the proposed
MIVPG improves Q-former in main VL tasks.

本研究探讨了多模态大型语言模型的性能，通过利用视觉适配器将视觉表示与语言模型相结合，在多个视觉语言任务中取得了最先进的表现。研究还提出了一种通用组件，名为多示例视觉提示生成器（MIVPG），通过利用图像或样本相同的补丁之间的实例相关性，将丰富的视觉表示合并到语言模型中。对来自不同场景的三个公共视觉 - 语言数据集进行定量评估，结果表明所提出的 MIVPG 可以改善主要的视觉 - 语言任务中的 Q-former 模型。