The significant breakthroughs of Medical Multi-Modal Large Language Models
(Med-MLLMs) renovate modern healthcare with robust information synthesis and
medical decision support. However, these models are often evaluated on
benchmarks that are unsuitable for the Med-MLLMs due to the intricate nature of
the real-world diagnostic frameworks, which encompass diverse medical
specialties and involve complex clinical decisions. Moreover, these benchmarks
are susceptible to data leakage, since Med-MLLMs are trained on large
assemblies of publicly available data. Thus, an isolated and clinically
representative benchmark is highly desirable for credible Med-MLLMs evaluation.
To this end, we introduce Asclepius, a novel Med-MLLM benchmark that rigorously
and comprehensively assesses model capability in terms of: distinct medical
specialties (cardiovascular, gastroenterology, etc.) and different diagnostic
capacities (perception, disease analysis, etc.). Grounded in 3 proposed core
principles, Asclepius ensures a comprehensive evaluation by encompassing 15
medical specialties, stratifying into 3 main categories and 8 sub-categories of
clinical tasks, and exempting from train-validate contamination. We further
provide an in-depth analysis of 6 Med-MLLMs and compare them with 5 human
specialists, providing insights into their competencies and limitations in
various medical contexts. Our work not only advances the understanding of
Med-MLLMs' capabilities but also sets a precedent for future evaluations and
the safe deployment of these models in clinical environments. We launch and
maintain a leaderboard for community assessment of Med-MLLM capabilities
(this https URL).

通过 Asclepius 作为新的 Med-MLLM 评估基准，全面评估 15 个医学专业、3 个主要临床任务类别和 8 个子类别的临床任务，以及 6 个 Med-MLLMs 和 5 个人类专家的能力和局限，推动对于 Med-MLLMs 能力的理解，并为未来评估和临床环境中安全使用这些模型奠定了先例。

Asclepius：医学多模态大型语言模型的频谱评估基准

Asclepius: A Spectrum Evaluation Benchmark for Medical Multi-Modal Large  Language Models

This paper gives an overview of the latest applications of Large Language
Models (LLMs) in the healthcare sector, highlighting their transformative role
in enhancing medical care quality. By processing vast amounts of data from
diverse medical domains, LLMs have become pivotal in assisting doctors,
healthcare providers, and patients. We explore their utilization in various
medical specialties, such as cancer diagnostics, dentistry, nephrology,
dermatology, etc. The paper includes the LLM methodologies applied in various
medical specialties, different data types in the medical domains and the
relevant input formatting for LLMs, along with practical use-cases of LLMs in
the healthcare domain.

该研究概述了大型语言模型（LLMs）在医疗保健领域的最新应用，突出了它们在提高医疗质量方面的变革性作用。通过处理大量来自不同医疗领域的数据，LLMs 已成为协助医生、医疗提供者和患者的关键要素。我们探讨了它们在癌症诊断、牙科学、肾脏病学、皮肤科等各种医学专业中的应用，并介绍了在不同医学领域的 LLMs 方法学、数据类型和适用于 LLMs 的相关输入格式，以及医疗领域中 LLMs 的实际应用案例。