The significant breakthroughs of Medical Multi-Modal Large Language Models
(Med-MLLMs) renovate modern healthcare with robust information synthesis and
medical decision support. However, these models are often evaluated on
benchmarks that are unsuitable for the Med-MLLMs due to the intricate nature of
the real-world diagnostic frameworks, which encompass diverse medical
specialties and involve complex clinical decisions. Moreover, these benchmarks
are susceptible to data leakage, since Med-MLLMs are trained on large
assemblies of publicly available data. Thus, an isolated and clinically
representative benchmark is highly desirable for credible Med-MLLMs evaluation.
To this end, we introduce Asclepius, a novel Med-MLLM benchmark that rigorously
and comprehensively assesses model capability in terms of: distinct medical
specialties (cardiovascular, gastroenterology, etc.) and different diagnostic
capacities (perception, disease analysis, etc.). Grounded in 3 proposed core
principles, Asclepius ensures a comprehensive evaluation by encompassing 15
medical specialties, stratifying into 3 main categories and 8 sub-categories of
clinical tasks, and exempting from train-validate contamination. We further
provide an in-depth analysis of 6 Med-MLLMs and compare them with 5 human
specialists, providing insights into their competencies and limitations in
various medical contexts. Our work not only advances the understanding of
Med-MLLMs' capabilities but also sets a precedent for future evaluations and
the safe deployment of these models in clinical environments. We launch and
maintain a leaderboard for community assessment of Med-MLLM capabilities
(this https URL).

通过 Asclepius 作为新的 Med-MLLM 评估基准，全面评估 15 个医学专业、3 个主要临床任务类别和 8 个子类别的临床任务，以及 6 个 Med-MLLMs 和 5 个人类专家的能力和局限，推动对于 Med-MLLMs 能力的理解，并为未来评估和临床环境中安全使用这些模型奠定了先例。

Asclepius：医学多模态大型语言模型的频谱评估基准

Asclepius: A Spectrum Evaluation Benchmark for Medical Multi-Modal Large  Language Models

Breast ultrasound (BUS) is a critical diagnostic tool in the field of breast
imaging, aiding in the early detection and characterization of breast
abnormalities. Interpreting breast ultrasound images commonly involves creating
comprehensive medical reports, containing vital information to promptly assess
the patient's condition. However, the ultrasound imaging system necessitates
capturing multiple images of various parts to compile a single report,
presenting a time-consuming challenge. To address this problem, we propose the
integration of multiple image analysis tools through a LangChain using Large
Language Models (LLM), into the breast reporting process. Through a combination
of designated tools and text generation through LangChain, our method can
accurately extract relevant features from ultrasound images, interpret them in
a clinical context, and produce comprehensive and standardized reports. This
approach not only reduces the burden on radiologists and healthcare
professionals but also enhances the consistency and quality of reports. The
extensive experiments shows that each tools involved in the proposed method can
offer qualitatively and quantitatively significant results. Furthermore,
clinical evaluation on the generated reports demonstrates that the proposed
method can make report in clinically meaningful way.

通过 LangChain 和大型语言模型 (Large Language Models, LLM) 的组合，我们提出了一种整合多种图像分析工具的方法，可以准确提取超声图像的相关特征，将其在临床背景中进行解释，并生成全面且标准化的报告，从而减轻放射科医师和医疗保健专业人员的负担，提高报告的一致性和质量。大量实验证明了所提出方法中涉及的每个工具均能提供定性和定量上显著的结果。此外，对所生成报告的临床评估表明，所提出的方法可以以临床意义的方式进行报告生成。

利用 LangChain 生成乳腺超声报告

Breast Ultrasound Report Generation using LangChain

There is an important need for methods to reduce radiation dose and imaging
time in myocardial perfusion imaging (MPI) SPECT. Deep learning (DL) methods
have demonstrated promise in predicting normal-count images from low-count
images for MPI SPECT, but the methods that have been objectively evaluated on
the clinical task of detecting perfusion defects have not shown improved
performance compared with low-count images. To address this need, we build upon
concepts from model-observer theory and our understanding of the human visual
system to propose a Detection task-specific DL-based approach for denoising MPI
SPECT images (DEMIST). The approach, while performing denoising, is designed to
preserve features that are known to impact observer performance on detection
tasks. We objectively evaluated the proposed method on the task of detecting
perfusion defects using a retrospective study with anonymized clinical data in
patients who underwent MPI studies (N = 338). Performance on the task of
detecting perfusion defects was quantified with an anthropomorphic channelized
Hotelling observer. Images denoised with DEMIST yielded significantly improved
detection performance compared to the corresponding low-dose images and images
denoised with a commonly used task-agnostic DL-based denoising method. Similar
results were observed with stratified analysis based on patient sex and defect
type. Additionally, the proposed method significantly improved performance
compared to the low-dose images in terms of the task-agnostic metrics of root
mean squared error and structural similarity index metric. A mathematical
analysis reveals that DEMIST preserves detection-task-specific features while
improving the noise properties, thus resulting in improved observer
performance. The results provide strong evidence for further clinical
evaluation of DEMIST to denoise low-count images in MPI SPECT.

通过基于深度学习的、对 MPI SPECT 进行去噪的特定检测任务方法（DEMIST）的客观评估，本研究提出的方法在检测心肌灌注缺损方面的表现显著优于对应的低剂量图像和普通的基于深度学习的去噪方法，并且该方法还在保持特定检测任务特征的同时改善了噪声特性。