Chest X-ray images are commonly used for predicting acute and chronic
cardiopulmonary conditions, but efforts to integrate them with structured
clinical data face challenges due to incomplete electronic health records
(EHR). This paper introduces \textbf{MedPromptX}, the first model to integrate
multimodal large language models (MLLMs), few-shot prompting (FP) and visual
grounding (VG) to combine imagery with EHR data for chest X-ray diagnosis. A
pre-trained MLLM is utilized to complement the missing EHR information,
providing a comprehensive understanding of patients' medical history.
Additionally, FP reduces the necessity for extensive training of MLLMs while
effectively tackling the issue of hallucination. Nevertheless, the process of
determining the optimal number of few-shot examples and selecting high-quality
candidates can be burdensome, yet it profoundly influences model performance.
Hence, we propose a new technique that dynamically refines few-shot data for
real-time adjustment to new patient scenarios. Moreover, VG aids in focusing
the model's attention on relevant regions of interest in X-ray images,
enhancing the identification of abnormalities. We release MedPromptX-VQA, a new
in-context visual question answering dataset encompassing interleaved image and
EHR data derived from MIMIC-IV and MIMIC-CXR databases. Results demonstrate the
SOTA performance of MedPromptX, achieving an 11% improvement in F1-score
compared to the baselines. Code and data are available at
https://github.com/BioMedIA-MBZUAI/MedPromptX.

利用多模态大型语言模型、少样本训练提示、视觉定位将影像与电子病历数据结合，该研究提出了 MedPromptX 模型，用于胸部 X 射线诊断，有效提高了异常的识别能力。

MedPromptX: 融入视听提示的胸部 X 射线诊断

MedPromptX: Grounded Multimodal Prompting for Chest X-ray Diagnosis

Advances in artificial intelligence (AI) have achieved expert-level
performance in medical imaging applications. Notably, self-supervised
vision-language foundation models can detect a broad spectrum of pathologies
without relying on explicit training annotations. However, it is crucial to
ensure that these AI models do not mirror or amplify human biases, thereby
disadvantaging historically marginalized groups such as females or Black
patients. The manifestation of such biases could systematically delay essential
medical care for certain patient subgroups. In this study, we investigate the
algorithmic fairness of state-of-the-art vision-language foundation models in
chest X-ray diagnosis across five globally-sourced datasets. Our findings
reveal that compared to board-certified radiologists, these foundation models
consistently underdiagnose marginalized groups, with even higher rates seen in
intersectional subgroups, such as Black female patients. Such demographic
biases present over a wide range of pathologies and demographic attributes.
Further analysis of the model embedding uncovers its significant encoding of
demographic information. Deploying AI systems with these biases in medical
imaging can intensify pre-existing care disparities, posing potential
challenges to equitable healthcare access and raising ethical questions about
their clinical application.

人工智能在医学影像应用中取得了专家级性能。然而，研究发现目前的视觉语言基础模型在胸部 X 射线诊断中普遍存在算法公平性问题，导致偏见对待历史上被边缘化的群体，并延迟特定人群的重要医疗护理，严重加剧现有的护理差异，对公平医疗保障带来挑战。

医学影像中专家级视觉语言基础模型的人口统计偏见

Demographic Bias of Expert-Level Vision-Language Foundation Models in  Medical Imaging

In the field of chest X-ray (CXR) diagnosis, existing works often focus
solely on determining where a radiologist looks, typically through tasks such
as detection, segmentation, or classification. However, these approaches are
often designed as black-box models, lacking interpretability. In this paper, we
introduce a novel and unified controllable interpretable pipeline for decoding
the intense focus of radiologists in CXR diagnosis. Our approach addresses
three key questions: where a radiologist looks, how long they focus on specific
areas, and what findings they diagnose. By capturing the intensity of the
radiologist's gaze, we provide a unified solution that offers insights into the
cognitive process underlying radiological interpretation. Unlike current
methods that rely on black-box machine learning models, which can be prone to
extracting erroneous information from the entire input image during the
diagnosis process, we tackle this issue by effectively masking out irrelevant
information. Our approach leverages a vision-language model, allowing for
precise control over the interpretation process while ensuring the exclusion of
irrelevant features. To train our model, we utilize an eye gaze dataset to
extract anatomical gaze information and generate ground truth heatmaps. Through
extensive experimentation, we demonstrate the efficacy of our method. We
showcase that the attention heatmaps, designed to mimic radiologists' focus,
encode sufficient and relevant information, enabling accurate classification
tasks using only a portion of CXR.

本文提出了一种新颖且统一的可控解释性流程，用于解码胸部 X 射线（CXR）诊断中放射科医师的密集关注点，通过捕捉放射科医师凝视的强度，提供了对放射学解释过程的认知过程的洞察。

解码放射科医生专注于准确的胸部 X 光诊断：一种可控和可解释的人工智能系统

Decoding Radiologists Intense Focus for Accurate CXR Diagnoses: A  Controllable and Interpretable AI System

While many studies have assessed the fairness of AI algorithms in the medical
field, the causes of differences in prediction performance are often unknown.
This lack of knowledge about the causes of bias hampers the efficacy of bias
mitigation, as evidenced by the fact that simple dataset balancing still often
performs best in reducing performance gaps but is unable to resolve all
performance differences. In this work, we investigate the causes of gender bias
in machine learning-based chest X-ray diagnosis. In particular, we explore the
hypothesis that breast tissue leads to underexposure of the lungs and causes
lower model performance. Methodologically, we propose a new sampling method
which addresses the highly skewed distribution of recordings per patient in two
widely used public datasets, while at the same time reducing the impact of
label errors. Our comprehensive analysis of gender differences across diseases,
datasets, and gender representations in the training set shows that dataset
imbalance is not the sole cause of performance differences. Moreover, relative
group performance differs strongly between datasets, indicating important
dataset-specific factors influencing male/female group performance. Finally, we
investigate the effect of breast tissue more specifically, by cropping out the
breasts from recordings, finding that this does not resolve the observed
performance gaps. In conclusion, our results indicate that dataset-specific
factors, not fundamental physiological differences, are the main drivers of
male--female performance gaps in chest X-ray analyses on widely used NIH and
CheXpert Dataset.

通过研究胸部 X 射线诊断中的性别偏见，提出了新的采样方法研究导致性别差异的因素，指出数据集特定因素是影响男女性能差距的主要驱动力。