Vision-language models have been widely explored across a wide range of tasks and achieve satisfactory performance. However, it's under-explored how to consolidate entity understanding through a varying number of images and to align it with the pre-trained language models for generative tasks. In this paper, we propose MIVC, a general multiple instance visual component to bridge the gap between various image inputs with off-the-shelf vision-language models by aggregating visual representations in a permutation-invariant fashion through a neural network. We show that MIVC could be plugged into the visual-language models to improve the model performance consistently on visual question answering, classification and captioning tasks on a public available e-commerce dataset with multiple images per product. Furthermore, we show that the component provides insight into the contribution of each image to the downstream tasks.

本文提出了MIVC，一种通用的多实例可视化组件，通过神经网络以排列不变的方式汇总视觉表示，以填补各种图像输入与预训练语言模型之间的差距。我们展示了MIVC如何整合到视觉语言模型中，显著提高了视觉问答、分类和标题任务的模型性能，并使用公共可用的电子商务数据集上的每个产品的多个图像展示了该组件对下游任务的贡献的见解。

MIVC：用于视觉语言模型的多实例视觉组件