The rapid advancement of large-scale vision-language models has showcased
remarkable capabilities across various tasks. However, the lack of extensive
and high-quality image-text data in medicine has greatly hindered the
development of large-scale medical vision-language models. In this work, we
present a diagnosis-guided bootstrapping strategy that exploits both image and
label information to construct vision-language datasets. Based on the
constructed dataset, we developed MedDr, a generalist foundation model for
healthcare capable of handling diverse medical data modalities, including
radiology, pathology, dermatology, retinography, and endoscopy. Moreover,
during inference, we propose a simple but effective retrieval-augmented medical
diagnosis strategy, which enhances the model's generalization ability.
Extensive experiments on visual question answering, medical report generation,
and medical image diagnosis demonstrate the superiority of our method.

利用诊断指导的引导策略和图像与标签信息，构建视觉 - 语言数据集，从而开发了一个通用的医疗基础模型 MedDr，能够处理多样化的医疗数据模态，包括放射学、病理学、皮肤科、眼底摄影和内窥镜检查，并提出了一种简单而有效的检索增强医学诊断策略，提高了模型的泛化能力，广泛的实验验证了我们方法的优越性。