Large Vision-Language Models (LVLMs) have shown significant potential in
assisting medical diagnosis by leveraging extensive biomedical datasets.
However, the advancement of medical image understanding and reasoning
critically depends on building high-quality visual instruction data, which is
costly and labor-intensive to obtain, particularly in the medical domain. To
mitigate this data-starving issue, we introduce Self-Training Large Language
and Vision Assistant for Medical (STLLaVA-Med). The proposed method is designed
to train a policy model (an LVLM) capable of auto-generating medical visual
instruction data to improve data efficiency, guided through Direct Preference
Optimization (DPO). Specifically, a more powerful and larger LVLM (e.g.,
GPT-4o) is involved as a biomedical expert to oversee the DPO fine-tuning
process on the auto-generated data, encouraging the policy model to align
efficiently with human preferences. We validate the efficacy and data
efficiency of STLLaVA-Med across three major medical Visual Question Answering
(VQA) benchmarks, demonstrating competitive zero-shot performance with the
utilization of only 9% of the medical data.

通过引入自我训练的大型语言和视觉助手 (STLLaVA-Med)，使用 LVLM 和直接优化原则 (DPO) 自动产生医学视觉指导数据，以提高数据效率，并且证实了在使用只有 9% 的医学数据的情况下，STLLaVA-Med 在三个主要的医学视觉问答 (VQA) 基准测试中具有竞争力的零测性能。

STLLaVA-Med：医学领域的自我训练大型语言与视觉助理

STLLaVA-Med: Self-Training Large Language and Vision Assistant for  Medical

This paper introduces MM-Instruct, a large-scale dataset of diverse and
high-quality visual instruction data designed to enhance the
instruction-following capabilities of large multimodal models (LMMs). While
existing visual instruction datasets often focus on question-answering, they
struggle to generalize to broader application scenarios such as creative
writing, summarization, or image analysis. To address these limitations, we
propose a novel approach to constructing MM-Instruct that leverages the strong
instruction-following capabilities of existing LLMs to generate novel visual
instruction data from large-scale but conventional image captioning datasets.
MM-Instruct first leverages ChatGPT to automatically generate diverse
instructions from a small set of seed instructions through augmenting and
summarization. It then matches these instructions with images and uses an
open-sourced large language model (LLM) to generate coherent answers to the
instruction-image pairs. The LLM is grounded by the detailed text descriptions
of images in the whole answer generation process to guarantee the alignment of
the instruction data. Moreover, we introduce a benchmark based on the generated
instruction data to evaluate the instruction-following capabilities of existing
LMMs. We demonstrate the effectiveness of MM-Instruct by training a LLaVA-1.5
model on the generated data, denoted as LLaVA-Instruct, which exhibits
significant improvements in instruction-following capabilities compared to
LLaVA-1.5 models. The MM-Instruct dataset, benchmark, and pre-trained models
are available at this https URL

该研究介绍了 MM-Instruct，这是一个大规模、多样化和高质量的视觉指导数据集，旨在增强大型多模态模型（LMMs）的指令跟随能力，并通过使用现有的 LLMs 从大规模图像字幕数据集生成新的视觉指导数据，并介绍了一个基于生成的指导数据来评估现有 LMMs 的指令跟随能力的基准。