Multi-modal Large Language Models (MLLMs) are increasingly prominent in the
field of artificial intelligence. Visual instruction fine-tuning (IFT) is a
vital process for aligning MLLMs' output with user's intentions. High-quality
and diversified instruction following data is the key to this fine-tuning
process. Recent studies propose to construct visual IFT datasets through a
multifaceted approach: transforming existing datasets with rule-based
templates, employing GPT-4 for rewriting annotations, and utilizing GPT-4V for
visual dataset pseudo-labeling. LLaVA-1.5 adopted similar approach and
construct LLaVA-mix-665k, which is one of the simplest, most widely used, yet
most effective IFT datasets today. Notably, when properly fine-tuned with this
dataset, MLLMs can achieve state-of-the-art performance on several benchmarks.
However, we noticed that models trained with this dataset often struggle to
follow user instructions properly in multi-round dialog. In addition, tradition
caption and VQA evaluation benchmarks, with their closed-form evaluation
structure, are not fully equipped to assess the capabilities of modern
open-ended generative MLLMs. This problem is not unique to the LLaVA-mix-665k
dataset, but may be a potential issue in all IFT datasets constructed from
image captioning or VQA sources, though the extent of this issue may vary. We
argue that datasets with diverse and high-quality detailed instruction
following annotations are essential and adequate for MLLMs IFT. In this work,
we establish a new IFT dataset, with images sourced from the COCO dataset along
with more diverse instructions. Our experiments show that when fine-tuned with
out proposed dataset, MLLMs achieve better performance on open-ended evaluation
benchmarks in both single-round and multi-round dialog setting.

利用 COOC 数据集和更多多样化指令建立了新的多模态大语言模型指令细调数据集，实验证明用该数据集进行的细调在单轮和多轮对话设置中，在开放式评估基准上能够获得更好的性能。