GPT-4 demonstrates high accuracy in medical QA tasks, leading with an
accuracy of 86.70%, followed by Med-PaLM 2 at 86.50%. However, around 14% of
errors remain. Additionally, current works use GPT-4 to only predict the
correct option without providing any explanation and thus do not provide any
insight into the thinking process and reasoning used by GPT-4 or other LLMs.
Therefore, we introduce a new domain-specific error taxonomy derived from
collaboration with medical students. Our GPT-4 USMLE Error (G4UE) dataset
comprises 4153 GPT-4 correct responses and 919 incorrect responses to the
United States Medical Licensing Examination (USMLE) respectively. These
responses are quite long (258 words on average), containing detailed
explanations from GPT-4 justifying the selected option. We then launch a
large-scale annotation study using the Potato annotation platform and recruit
44 medical experts through Prolific, a well-known crowdsourcing platform. We
annotated 300 out of these 919 incorrect data points at a granular level for
different classes and created a multi-label span to identify the reasons behind
the error. In our annotated dataset, a substantial portion of GPT-4's incorrect
responses is categorized as a "Reasonable response by GPT-4," by annotators.
This sheds light on the challenge of discerning explanations that may lead to
incorrect options, even among trained medical professionals. We also provide
medical concepts and medical semantic predications extracted using the SemRep
tool for every data point. We believe that it will aid in evaluating the
ability of LLMs to answer complex medical questions. We make the resources
available at this https URL .

GPT-4 在医疗问答任务中表现出高准确性，但仍存在约 14% 的错误。因此，我们引入了一种与医学学生合作得出的新的领域特定错误分类系统。我们的 GPT-4 USMLE Error（G4UE）数据集包含 4153 个 GPT-4 正确回答和 919 个不正确回答的美国医学执照考试（USMLE）问题。这些回答较长（平均 258 个单词），包含了 GPT-4 解释所选选项的详细说明。通过大规模的注释研究和医学专家的参与，我们对其中 300 个不正确的数据点进行了细致的分类注释，以确定错误背后的原因。我们的注释数据集中，相当一部分 GPT-4 的不正确回答被注释为 “GPT-4 合理回答”，这揭示了在受过训练的医学专业人士中，辨别可能导致错误选项的解释所面临的挑战。我们还提供了使用 SemRep 工具提取的医学概念和医学语义预测，这将有助于评估语言模型对复杂医学问题的回答能力。我们将这些资源提供在指定的网址。