Multi-modal Large Language Models (MLLMs) have shown remarkable capabilities
in many vision-language tasks. Nevertheless, most MLLMs still lack the
Referential Comprehension (RC) ability to identify a specific object or area in
images, limiting their application in fine-grained perception tasks. This paper
proposes a novel method to enhance the RC capability for MLLMs. Our model
represents the referring object in the image using the coordinates of its
bounding box and converts the coordinates into texts in a specific format. This
allows the model to treat the coordinates as natural language. Moreover, we
construct the instruction tuning dataset with various designed RC tasks at a
low cost by unleashing the potential of annotations in existing datasets. To
further boost the RC ability of the model, we propose a self-consistent
bootstrapping method that extends dense object annotations of a dataset into
high-quality referring-expression-bounding-box pairs. The model is trained
end-to-end with a parameter-efficient tuning framework that allows both
modalities to benefit from multi-modal instruction tuning. This framework
requires fewer trainable parameters and less training data. Experimental
results on conventional vision-language and RC tasks demonstrate the superior
performance of our method. For instance, our model exhibits a 12.0% absolute
accuracy improvement over Instruct-BLIP on VSR and surpasses Kosmos-2 by 24.7%
on RefCOCO_val under zero-shot settings. We also attain the top position on the
leaderboard of MMBench. The models, datasets, and codes are publicly available
at this https URL

该论文提出了一种新的方法来增强多模态大语言模型的指称理解能力，通过使用边界框的坐标表示图像中的指称对象，并将其转化为特定格式的文本，使模型能够以自然语言处理坐标，同时通过自洽引导方法和参数高效的调整框架进一步提升指称理解能力。实验结果表明，该方法在传统的视觉语言和指称理解任务中的性能优于其他方法。