As opposed to evaluating computation and logic-based reasoning, current
bench2 marks for evaluating large language models (LLMs) in medicine are
primarily focused on question-answering involving domain knowledge and
descriptive rea4 soning. While such qualitative capabilities are vital to
medical diagnosis, in real5 world scenarios, doctors frequently use clinical
calculators that follow quantitative equations and rule-based reasoning
paradigms for evidence-based decision support. To this end, we propose
MedCalc-Bench, a first-of-its-kind dataset focused on evaluating the medical
calculation capability of LLMs. MedCalc-Bench contains an evaluation set of
over 1000 manually reviewed instances from 55 different medical calculation
tasks. Each instance in MedCalc-Bench consists of a patient note, a question
requesting to compute a specific medical value, a ground truth answer, and a
step-by-step explanation showing how the answer is obtained. While our
evaluation results show the potential of LLMs in this area, none of them are
effective enough for clinical settings. Common issues include extracting the
incorrect entities, not using the correct equation or rules for a calculation
task, or incorrectly performing the arithmetic for the computation. We hope our
study highlights the quantitative knowledge and reasoning gaps in LLMs within
medical settings, encouraging future improvements of LLMs for various clinical
calculation tasks.

评估医学中的大型语言模型 (LLMs) 时，与计算和基于逻辑的推理相反，当前的基准测试主要集中在涉及领域知识和描述性推理的问答上。然而，在现实世界的情况下，医生经常使用遵循定量方程和基于规则推理范例的临床计算器作为基于证据的决策支持。为此，我们提出了 MedCalc-Bench，这是一个首个针对评估 LLMs 在医学计算能力方面的数据集。MedCalc-Bench 包含了来自 55 个不同医学计算任务的 1000 多个手动审核实例的评估集。每个 MedCalc-Bench 实例包括患者记录、请求计算特定医学数值的问题、基本真相答案和逐步说明显示如何获取答案。尽管我们的评估结果显示 LLMs 在这个领域具有潜力，但它们还没有在临床环境中足够有效。常见问题包括提取错误的实体、不使用正确的方程式或规则进行计算任务，或错误地执行计算的算术操作。我们希望我们的研究能够突出 LLMs 在医学环境中的定量知识和推理差距，促进未来改进 LLMs 以用于各种临床计算任务。