Current medical artificial intelligence systems are often limited to narrow
applications, hindering their widespread adoption in clinical practice. To
address this limitation, we propose MedVersa, a generalist learner that enables
flexible learning and tasking for medical image interpretation. By leveraging a
large language model as a learnable orchestrator, MedVersa can learn from both
visual and linguistic supervision, support multimodal inputs, and perform
real-time task specification. This versatility allows MedVersa to adapt to
various clinical scenarios and perform multifaceted medical image analysis. We
introduce MedInterp, the largest multimodal dataset to date for medical image
interpretation, consisting of over 13 million annotated instances spanning 11
tasks across 3 modalities, to support the development of MedVersa. Our
experiments demonstrate that MedVersa achieves state-of-the-art performance in
9 tasks, sometimes outperforming specialist counterparts by over 10%. MedVersa
is the first to showcase the viability of multimodal generative medical AI in
implementing multimodal outputs, inputs, and dynamic task specification,
highlighting its potential as a multifunctional system for comprehensive
medical image analysis. This generalist approach to medical image
interpretation paves the way for more adaptable and efficient AI-assisted
clinical decision-making.

提出了一种名为 MedVersa 的医学图像解释通用学习器，通过利用大型语言模型作为可学习的协调器，支持视觉和语言监督学习以及多模态输入，在医学图像解释领域得到了最先进的性能，在临床决策辅助方面具有潜力。