Multimodal artificial intelligence (AI) systems have the potential to enhance
clinical decision-making by interpreting various types of medical data.
However, the effectiveness of these models across all medical fields is
uncertain. Each discipline presents unique challenges that need to be addressed
for optimal performance. This complexity is further increased when attempting
to integrate different fields into a single model. Here, we introduce an
alternative approach to multimodal medical AI that utilizes the generalist
capabilities of a large language model (LLM) as a central reasoning engine.
This engine autonomously coordinates and deploys a set of specialized medical
AI tools. These tools include text, radiology and histopathology image
interpretation, genomic data processing, web searches, and document retrieval
from medical guidelines. We validate our system across a series of clinical
oncology scenarios that closely resemble typical patient care workflows. We
show that the system has a high capability in employing appropriate tools
(97%), drawing correct conclusions (93.6%), and providing complete (94%), and
helpful (89.2%) recommendations for individual patient cases while consistently
referencing relevant literature (82.5%) upon instruction. This work provides
evidence that LLMs can effectively plan and execute domain-specific models to
retrieve or synthesize new information when used as autonomous agents. This
enables them to function as specialist, patient-tailored clinical assistants.
It also simplifies regulatory compliance by allowing each component tool to be
individually validated and approved. We believe, that our work can serve as a
proof-of-concept for more advanced LLM-agents in the medical domain.

多模态人工智能系统能够通过解释各种类型的医疗数据来增强临床决策，本研究引入了一种利用大型语言模型作为中央推理引擎的多模态医疗人工智能的替代方法，通过验证该系统在临床肿瘤学场景中的表现，证实了语言模型能够作为专科、以患者为中心的临床助手的有效性。