Medical artificial general intelligence (MAGI) enables one foundation model
to solve different medical tasks, which is very practical in the medical
domain. It can significantly reduce the requirement of large amounts of
task-specific data by sufficiently sharing medical knowledge among different
tasks. However, due to the challenges of designing strongly generalizable
models with limited and complex medical data, most existing approaches tend to
develop task-specific models. To take a step towards MAGI, we propose a new
paradigm called Medical-knOwledge-enhanced mulTimOdal pretRaining (MOTOR). In
MOTOR, we combine two kinds of basic medical knowledge, i.e., general and
specific knowledge, in a complementary manner to boost the general pretraining
process. As a result, the foundation model with comprehensive basic knowledge
can learn compact representations from pretraining radiographic data for better
cross-modal alignment. MOTOR unifies the understanding and generation, which
are two kinds of core intelligence of an AI system, into a single medical
foundation model, to flexibly handle more diverse medical tasks. To enable a
comprehensive evaluation and facilitate further research, we construct a
medical multimodal benchmark including a wide range of downstream tasks, such
as chest x-ray report generation and medical visual question answering.
Extensive experiments on our benchmark show that MOTOR obtains promising
results through simple task-oriented adaptation. The visualization shows that
the injected knowledge successfully highlights key information in the medical
data, demonstrating the excellent interpretability of MOTOR. Our MOTOR
successfully mimics the human practice of fulfilling a "medical student" to
accelerate the process of becoming a "specialist". We believe that our work
makes a significant stride in realizing MAGI.

提出了一种基于 Medical-knOwledge-enhanced mulTimOdal pretRaining (MOTOR) 模型的医学人工通用智能模型，通过融合不同的基本医学知识，实现了对于医学预训练模型较好的跨模态对齐，形成了一种医疗智能基础模型，引入指导性知识在下游任务中展示了优异的可解释性和潜力。