Large Language Models (LLMs) have the potential of facilitating the
development of Artificial Intelligence technology to assist medical experts for
interactive decision support, which has been demonstrated by their competitive
performances in Medical QA. However, while impressive, the required quality bar
for medical applications remains far from being achieved. Currently, LLMs
remain challenged by outdated knowledge and by their tendency to generate
hallucinated content. Furthermore, most benchmarks to assess medical knowledge
lack reference gold explanations which means that it is not possible to
evaluate the reasoning of LLMs predictions. Finally, the situation is
particularly grim if we consider benchmarking LLMs for languages other than
English which remains, as far as we know, a totally neglected topic. In order
to address these shortcomings, in this paper we present MedExpQA, the first
multilingual benchmark based on medical exams to evaluate LLMs in Medical
Question Answering. To the best of our knowledge, MedExpQA includes for the
first time reference gold explanations written by medical doctors which can be
leveraged to establish various gold-based upper-bounds for comparison with LLMs
performance. Comprehensive multilingual experimentation using both the gold
reference explanations and Retrieval Augmented Generation (RAG) approaches show
that performance of LLMs still has large room for improvement, especially for
languages other than English. Furthermore, and despite using state-of-the-art
RAG methods, our results also demonstrate the difficulty of obtaining and
integrating readily available medical knowledge that may positively impact
results on downstream evaluations for Medical Question Answering. So far the
benchmark is available in four languages, but we hope that this work may
encourage further development to other languages.

该论文介绍了 MedExpQA，一个基于医学考试的多语言基准，用于评估大型语言模型在医学问答中的表现，并指出目前大型语言模型的性能还有很大的改进空间，特别是对于英语以外的语言。同时，该研究还强调了获取和整合可用的医学知识对于医学问答的后续评估结果具有困难，并呼吁进一步开发其他语言的基准。