Recent advancements in multimodal foundation models have showcased impressive
capabilities in understanding and reasoning with visual and textual
information. Adapting these foundation models trained for general usage to
specialized domains like biomedicine requires large-scale domain-specific
instruction datasets. While existing works have explored curating such datasets
automatically, the resultant datasets are not explicitly aligned with domain
expertise. In this work, we propose a data-centric framework, Biomedical Visual
Instruction Tuning with Clinician Preference Alignment (BioMed-VITAL), that
incorporates clinician preferences into both stages of generating and selecting
instruction data for tuning biomedical multimodal foundation models. First,
during the generation stage, we prompt the GPT-4V generator with a diverse set
of clinician-selected demonstrations for preference-aligned data candidate
generation. Then, during the selection phase, we train a separate selection
model, which explicitly distills clinician and policy-guided model preferences
into a rating function to select high-quality data for medical instruction
tuning. Results show that the model tuned with the instruction-following data
from our method demonstrates a significant improvement in open visual chat
(18.5% relatively) and medical VQA (win rate up to 81.73%). Our
instruction-following data and models are available at BioMed-VITAL.github.io.

本文提出了一种数据中心的框架 BioMed-VITAL，将临床医生的偏好纳入到生成和选择生物医学多模式基础模型调整的指导数据的过程中，并通过这种方法获得的指导数据进行调整后的模型在开放式视觉对话和医疗问答中表现出显著改进。