Medicine is inherently multimodal and multitask, with diverse data modalities spanning text, imaging. However, most models in medical field are unimodal single tasks and lack good generalizability and explainability. In this study, we introduce MedViLaM, a unified vision-language model towards a generalist model for medical data that can flexibly encode and interpret various forms of medical data, including clinical language and imaging, all using the same set of model weights. To facilitate the creation of such multi-task model, we have curated MultiMedBench, a comprehensive pretaining dataset and benchmark consisting of several distinct tasks, i.e., continuous question-answering, multi-label disease classification, disease localization, generation and summarization of radiology reports. MedViLaM demonstrates strong performance across all MultiMedBench tasks, frequently outpacing other generalist models by a significant margin. Additionally, we present instances of zero-shot generalization to new medical concepts and tasks, effective transfer learning across different tasks, and the emergence of zero-shot medical reasoning.

本研究解决了医疗领域单模态模型缺乏泛化性和可解释性的问题。提出的MedViLaM模型能够灵活编码和解释多种医疗数据，且在多个任务上表现优异，超越其他模型。此外，模型展示了在新医疗概念和任务上的零-shot 泛化能力，展现出其潜在的影响力。

MedViLaM：一种具有先进泛化性和可解释性的多模态大语言模型，用于医疗数据理解和生成