LLMs have demonstrated impressive performance in answering medical questions,
such as passing medical licensing examinations. However, most existing
benchmarks rely on board exam questions or general medical questions, falling
short in capturing the complexity of realistic clinical cases. Moreover, the
lack of reference explanations for answers hampers the evaluation of model
explanations, which are crucial to supporting doctors in making complex medical
decisions. To address these challenges, we construct two new datasets: JAMA
Clinical Challenge and Medbullets. JAMA Clinical Challenge consists of
questions based on challenging clinical cases, while Medbullets comprises USMLE
Step 2&3 style clinical questions. Both datasets are structured as
multiple-choice question-answering tasks, where each question is accompanied by
an expert-written explanation. We evaluate four LLMs on the two datasets using
various prompts. Experiments demonstrate that our datasets are harder than
previous benchmarks. The inconsistency between automatic and human evaluations
of model-generated explanations highlights the need to develop new metrics to
support future research on explainable medical QA.

通过构建两个新的数据集，利用多个评估指标以及医学专家编写的解释进行实验，我们发现 LLMs 在回答医学问题方面表现出色，但是现有的基准测试数据集在捕捉真实临床病例的复杂性以及提供参考解释方面存在不足，因此需要开发新的度量指标以支持可解释医疗问答的未来研究。

在回答和解释棘手的医学问题上评估大型语言模型

Benchmarking Large Language Models on Answering and Explaining  Challenging Medical Questions

Providing high quality explanations for AI predictions based on machine
learning is a challenging and complex task. To work well it requires, among
other factors: selecting a proper level of generality/specificity of the
explanation; considering assumptions about the familiarity of the explanation
beneficiary with the AI task under consideration; referring to specific
elements that have contributed to the decision; making use of additional
knowledge (e.g. expert evidence) which might not be part of the prediction
process; and providing evidence supporting negative hypothesis. Finally, the
system needs to formulate the explanation in a clearly interpretable, and
possibly convincing, way. Given these considerations, ANTIDOTE fosters an
integrated vision of explainable AI, where low-level characteristics of the
deep learning process are combined with higher level schemes proper of the
human argumentation capacity. ANTIDOTE will exploit cross-disciplinary
competences in deep learning and argumentation to support a broader and
innovative view of explainable AI, where the need for high-quality explanations
for clinical cases deliberation is critical. As a first result of the project,
we publish the Antidote CasiMedicos dataset to facilitate research on
explainable AI in general, and argumentation in the medical domain in
particular.

本文介绍了解释 AI 预测需要考虑的诸多因素，并阐述了 ANTIDOTE 项目利用深度学习和论证交叉学科知识支持解释 AI 的方法，为医疗领域的高质量解释提供了一种创新的视角。

HiTZ@Antidote：面向数字医疗的基于论证的可解释人工智能

HiTZ@Antidote: Argumentation-driven Explainable Artificial Intelligence  for Digital Medicine

In the event of stroke, a catheter-guided procedure (thrombectomy) is used to
remove blood clots. Feasibility of machine learning based automatic
classifications for thrombus detection on digital substraction angiography
(DSA) sequences has been demonstrated. It was however not used live in the
clinic, yet. We present an open-source tool for automatic thrombus
classification and test it on three selected clinical cases regarding
functionality and classification runtime. With our trained model all large
vessel occlusions in the M1 segment were correctly classified. One small
remaining M3 thrombus was not detected. Runtime was in the range from 1 to 10
seconds depending on the used hardware. We conclude that our open-source
software tool enables clinical staff to classify DSA sequences in (close to)
realtime and can be used for further studies in clinics.

本文介绍了一种开源工具，采用机器学习技术实现数字减影血管造影（DSA）血栓检测的自动分类，并在三个临床病例中进行了测试，其模型能够正确分类 M1 分段中的所有大血管闭塞，并可以在 (接近) 实时的情况下被临床工作人员使用。