Despite recent advances in the general visual instruction-following ability
of Multimodal Large Language Models (MLLMs), they still struggle with critical
problems when required to provide a precise and detailed response to a visual
instruction: (1) failure to identify novel objects or entities, (2) mention of
non-existent objects, and (3) neglect of object's attributed details. Intuitive
solutions include improving the size and quality of data or using larger
foundation models. They show effectiveness in mitigating these issues, but at
an expensive cost of collecting a vast amount of new data and introducing a
significantly larger model. Standing at the intersection of these approaches,
we examine the three object-oriented problems from the perspective of the
image-to-text mapping process by the multimodal connector. In this paper, we
first identify the limitations of multimodal connectors stemming from
insufficient training data. Driven by this, we propose to enhance the mapping
with retrieval-augmented tag tokens, which contain rich object-aware
information such as object names and attributes. With our Tag-grounded visual
instruction tuning with retrieval Augmentation (TUNA), we outperform baselines
that share the same language model and training data on 12 benchmarks.
Furthermore, we show the zero-shot capability of TUNA when provided with
specific datastores.

通过增强检索增强标记令牌，我们提出了 Tag-grounded visual instruction tuning with retrieval Augmentation（TUNA），在 12 个基准测试中优于使用相同语言模型和训练数据的基线模型，并展示了 TUNA 的零 - shot 能力。