This paper introduces MM-Instruct, a large-scale dataset of diverse and
high-quality visual instruction data designed to enhance the
instruction-following capabilities of large multimodal models (LMMs). While
existing visual instruction datasets often focus on question-answering, they
struggle to generalize to broader application scenarios such as creative
writing, summarization, or image analysis. To address these limitations, we
propose a novel approach to constructing MM-Instruct that leverages the strong
instruction-following capabilities of existing LLMs to generate novel visual
instruction data from large-scale but conventional image captioning datasets.
MM-Instruct first leverages ChatGPT to automatically generate diverse
instructions from a small set of seed instructions through augmenting and
summarization. It then matches these instructions with images and uses an
open-sourced large language model (LLM) to generate coherent answers to the
instruction-image pairs. The LLM is grounded by the detailed text descriptions
of images in the whole answer generation process to guarantee the alignment of
the instruction data. Moreover, we introduce a benchmark based on the generated
instruction data to evaluate the instruction-following capabilities of existing
LMMs. We demonstrate the effectiveness of MM-Instruct by training a LLaVA-1.5
model on the generated data, denoted as LLaVA-Instruct, which exhibits
significant improvements in instruction-following capabilities compared to
LLaVA-1.5 models. The MM-Instruct dataset, benchmark, and pre-trained models
are available at this https URL

该研究介绍了 MM-Instruct，这是一个大规模、多样化和高质量的视觉指导数据集，旨在增强大型多模态模型（LMMs）的指令跟随能力，并通过使用现有的 LLMs 从大规模图像字幕数据集生成新的视觉指导数据，并介绍了一个基于生成的指导数据来评估现有 LMMs 的指令跟随能力的基准。