Security concerns related to Large Language Models (LLMs) have been
extensively explored, yet the safety implications for Multimodal Large Language
Models (MLLMs), particularly in medical contexts (MedMLLMs), remain
insufficiently studied. This paper delves into the underexplored security
vulnerabilities of MedMLLMs, especially when deployed in clinical environments
where the accuracy and relevance of question-and-answer interactions are
critically tested against complex medical challenges. By combining existing
clinical medical data with atypical natural phenomena, we redefine two types of
attacks: mismatched malicious attack (2M-attack) and optimized mismatched
malicious attack (O2M-attack). Using our own constructed voluminous 3MAD
dataset, which covers a wide range of medical image modalities and harmful
medical scenarios, we conduct a comprehensive analysis and propose the MCM
optimization method, which significantly enhances the attack success rate on
MedMLLMs. Evaluations with this dataset and novel attack methods, including
white-box attacks on LLaVA-Med and transfer attacks on four other
state-of-the-art models, indicate that even MedMLLMs designed with enhanced
security features are vulnerable to security breaches. Our work underscores the
urgent need for a concerted effort to implement robust security measures and
enhance the safety and efficacy of open-source MedMLLMs, particularly given the
potential severity of jailbreak attacks and other malicious or clinically
significant exploits in medical settings. For further research and replication,
anonymous access to our code is available at
this https URL Warning: Medical large model
jailbreaking may generate content that includes unverified diagnoses and
treatment recommendations. Always consult professional medical advice.

介绍了 MedMLLMs（医学环境下的多模态大型语言模型）的安全漏洞，并提出了两种攻击类型（2M-attack 和 O2M-attack），并通过构建 3MAD 数据集进行评估，指出即使设计具有增强安全功能的 MedMLLMs 也存在安全漏洞，强调了在医疗环境中实施强大安全措施和增强开源 MedMLLMs 的安全性的紧迫性。

跨模态越狱与医疗多模态大语言模型的不匹配攻击

Cross-Modality Jailbreak and Mismatched Attacks on Medical Multimodal  Large Language Models

Diagnosing and managing a patient is a complex, sequential decision making
process that requires physicians to obtain information -- such as which tests
to perform -- and to act upon it. Recent advances in artificial intelligence
(AI) and large language models (LLMs) promise to profoundly impact clinical
care. However, current evaluation schemes overrely on static medical
question-answering benchmarks, falling short on interactive decision-making
that is required in real-life clinical work. Here, we present AgentClinic: a
multimodal benchmark to evaluate LLMs in their ability to operate as agents in
simulated clinical environments. In our benchmark, the doctor agent must
uncover the patient's diagnosis through dialogue and active data collection. We
present two open benchmarks: a multimodal image and dialogue environment,
AgentClinic-NEJM, and a dialogue-only environment, AgentClinic-MedQA. We embed
cognitive and implicit biases both in patient and doctor agents to emulate
realistic interactions between biased agents. We find that introducing bias
leads to large reductions in diagnostic accuracy of the doctor agents, as well
as reduced compliance, confidence, and follow-up consultation willingness in
patient agents. Evaluating a suite of state-of-the-art LLMs, we find that
several models that excel in benchmarks like MedQA are performing poorly in
AgentClinic-MedQA. We find that the LLM used in the patient agent is an
important factor for performance in the AgentClinic benchmark. We show that
both having limited interactions as well as too many interaction reduces
diagnostic accuracy in doctor agents. The code and data for this work is
publicly available at this https URL

诊断和治疗患者是一个复杂的、顺序的决策过程，医生需要获取信息并进行相应操作。本研究介绍了 AgentClinic，一个用于评估大型语言模型在模拟临床环境中扮演医生角色能力的多模态基准测试。

AgentClinic：评估 AI 在模拟临床环境中的多模态代理基准

AgentClinic: a multimodal agent benchmark to evaluate AI in simulated  clinical environments

This study introduces EHRNoteQA, a novel patient-specific question answering
benchmark tailored for evaluating Large Language Models (LLMs) in clinical
environments. Based on MIMIC-IV Electronic Health Record (EHR), a team of three
medical professionals has curated the dataset comprising 962 unique questions,
each linked to a specific patient's EHR clinical notes. What makes EHRNoteQA
distinct from existing EHR-based benchmarks is as follows: Firstly, it is the
first dataset to adopt a multi-choice question answering format, a design
choice that effectively evaluates LLMs with reliable scores in the context of
automatic evaluation, compared to other formats. Secondly, it requires an
analysis of multiple clinical notes to answer a single question, reflecting the
complex nature of real-world clinical decision-making where clinicians review
extensive records of patient histories. Our comprehensive evaluation on various
large language models showed that their scores on EHRNoteQA correlate more
closely with their performance in addressing real-world medical questions
evaluated by clinicians than their scores from other LLM benchmarks. This
underscores the significance of EHRNoteQA in evaluating LLMs for medical
applications and highlights its crucial role in facilitating the integration of
LLMs into healthcare systems. The dataset will be made available to the public
under PhysioNet credential access, promoting further research in this vital
field.

这项研究介绍了 EHRNoteQA，这是一个为评估临床环境中大型语言模型（LLMs）而定制的创新患者特定问题回答基准。它是首个采用多项选择问题回答格式的数据集，通过分析多个临床记录来回答单个问题，与其他格式相比，在自动评估中有效地评估 LLMs 的可靠性得分。在各种大型语言模型上进行的综合评估表明，它们在 EHRNoteQA 上的得分与通过临床医生评估的真实医学问题的表现更密切相关，凸显了 EHRNoteQA 在评估医学应用的 LLMs 中的重要性，以及它在促进 LLMs 融入医疗系统中的关键作用。该数据集将在 PhysioNet 凭据访问下向公众提供，促进这个重要领域的进一步研究。