Despite recent advances in the general visual instruction-following ability
of Multimodal Large Language Models (MLLMs), they still struggle with critical
problems when required to provide a precise and detailed response to a visual
instruction: (1) failure to identify novel objects or entities, (2) mention of
non-existent objects, and (3) neglect of object's attributed details. Intuitive
solutions include improving the size and quality of data or using larger
foundation models. They show effectiveness in mitigating these issues, but at
an expensive cost of collecting a vast amount of new data and introducing a
significantly larger model. Standing at the intersection of these approaches,
we examine the three object-oriented problems from the perspective of the
image-to-text mapping process by the multimodal connector. In this paper, we
first identify the limitations of multimodal connectors stemming from
insufficient training data. Driven by this, we propose to enhance the mapping
with retrieval-augmented tag tokens, which contain rich object-aware
information such as object names and attributes. With our Tag-grounded visual
instruction tuning with retrieval Augmentation (TUNA), we outperform baselines
that share the same language model and training data on 12 benchmarks.
Furthermore, we show the zero-shot capability of TUNA when provided with
specific datastores.

通过增强检索增强标记令牌，我们提出了 Tag-grounded visual instruction tuning with retrieval Augmentation（TUNA），在 12 个基准测试中优于使用相同语言模型和训练数据的基线模型，并展示了 TUNA 的零 - shot 能力。

通过检索的标签提醒多模态大型语言模型中的物体感知知识

Reminding Multimodal Large Language Models of Object-aware Knowledge  with Retrieved Tags

Despite vision-language models' (VLMs) remarkable capabilities as versatile
visual assistants, two substantial challenges persist within the existing VLM
frameworks: (1) lacking task diversity in pretraining and visual instruction
tuning, and (2) annotation error and bias in GPT-4 synthesized instruction
tuning data. Both challenges lead to issues such as poor generalizability,
hallucination, and catastrophic forgetting. To address these challenges, we
construct Vision-Flan, the most diverse publicly available visual instruction
tuning dataset to date, comprising 187 diverse tasks and 1,664,261 instances
sourced from academic datasets, and each task is accompanied by an
expert-written instruction. In addition, we propose a two-stage instruction
tuning framework, in which VLMs are firstly finetuned on Vision-Flan and
further tuned on GPT-4 synthesized data. We find this two-stage tuning
framework significantly outperforms the traditional single-stage visual
instruction tuning framework and achieves the state-of-the-art performance
across a wide range of multi-modal evaluation benchmarks. Finally, we conduct
in-depth analyses to understand visual instruction tuning and our findings
reveal that: (1) GPT-4 synthesized data does not substantially enhance VLMs'
capabilities but rather modulates the model's responses to human-preferred
formats; (2) A minimal quantity (e.g., 1,000) of GPT-4 synthesized data can
effectively align VLM responses with human-preference; (3) Visual instruction
tuning mainly helps large-language models (LLMs) to understand visual features.

通过构建最具多样性的 Vision-Flan 数据集和两阶段指导调优框架，本研究解决了现有 VLM 框架中的任务多样性和注释错误的挑战，并在广泛的多模态评估基准中取得了最新的最佳性能。