This paper introduces MedExQA, a novel benchmark in medical question-answering, to evaluate large language models' (LLMs) understanding of medical knowledge through explanations. By constructing datasets across five distinct medical specialties that are underrepresented in current datasets and further incorporating multiple explanations for each question-answer pair, we address a major gap in current medical QA benchmarks which is the absence of comprehensive assessments of LLMs' ability to generate nuanced medical explanations. Our work highlights the importance of explainability in medical LLMs, proposes an effective methodology for evaluating models beyond classification accuracy, and sheds light on one specific domain, speech language pathology, where current LLMs including GPT4 lack good understanding. Our results show generation evaluation with multiple explanations aligns better with human assessment, highlighting an opportunity for a more robust automated comprehension assessment for LLMs. To diversify open-source medical LLMs (currently mostly based on Llama2), this work also proposes a new medical model, MedPhi-2, based on Phi-2 (2.7B). The model outperformed medical LLMs based on Llama2-70B in generating explanations, showing its effectiveness in the resource-constrained medical domain. We will share our benchmark datasets and the trained model.

本研究介绍了MedExQA，这是一个用于评估大型语言模型对医学知识的理解能力的新型基准，通过构建涵盖五个不同医学专业的数据集，并且为每个问题-答案对提供多个解释，填补了当前医学问答基准的一个重要空白，即缺乏模型生成细致医学解释的全面评估。我们的工作强调了医学语言模型可解释性的重要性，提出了一个评估模型超越分类准确性的有效方法，并在特定领域——言语病理学中，揭示了当前GPT4等语言模型理解能力不足的问题。我们的结果表明，使用多个解释进行生成评估更符合人类评估结果，提供了一个更稳健的自动理解评估机制的机会。为了丰富开源的医学语言模型（目前主要基于Llama2），我们还提出了一种新的医学模型MedPhi-2，基于Phi-2 (2.7B)。该模型在生成解释方面的性能优于基于Llama2-70B的医学语言模型，展示了它在资源受限的医学领域的有效性。我们将分享我们的基准数据集和训练好的模型。

MedExQA：具备多重解释的医学问答基准