In recent years, Large Language Models (LLMs) have demonstrated an impressive
ability to encode knowledge during pre-training on large text corpora. They can
leverage this knowledge for downstream tasks like question answering (QA), even
in complex areas involving health topics. Considering their high potential for
facilitating clinical work in the future, understanding the quality of encoded
medical knowledge and its recall in LLMs is an important step forward. In this
study, we examine the capability of LLMs to exhibit medical knowledge recall by
constructing a novel dataset derived from systematic reviews -- studies
synthesizing evidence-based answers for specific medical questions. Through
experiments on the new MedREQAL dataset, comprising question-answer pairs
extracted from rigorous systematic reviews, we assess six LLMs, such as GPT and
Mixtral, analyzing their classification and generation performance. Our
experimental insights into LLM performance on the novel biomedical QA dataset
reveal the still challenging nature of this task.

近年来，大型语言模型（LLMs）在大文本语料库的预训练过程中展示了令人印象深刻的知识编码能力。本研究通过构建一个新的基于系统评述的数据集，评估了六种 LLMs（如 GPT 和 Mixtral）在医学知识回忆方面的能力，并揭示了此任务的复杂性。