Recent advancements in Large Language Models (LLMs) have demonstrated their
potential in delivering accurate answers to questions about world knowledge.
Despite this, existing benchmarks for evaluating LLMs in healthcare
predominantly focus on medical doctors, leaving other critical healthcare
professions underrepresented. To fill this research gap, we introduce the
Examinations for Medical Personnel in Chinese (EMPEC), a pioneering large-scale
healthcare knowledge benchmark in traditional Chinese. EMPEC consists of
157,803 exam questions across 124 subjects and 20 healthcare professions,
including underrepresented occupations like Optometrists and Audiologists. Each
question is tagged with its release time and source, ensuring relevance and
authenticity. We conducted extensive experiments on 17 LLMs, including
proprietary, open-source models, general domain models and medical specific
models, evaluating their performance under various settings. Our findings
reveal that while leading models like GPT-4 achieve over 75\% accuracy, they
still struggle with specialized fields and alternative medicine. Surprisingly,
general-purpose LLMs outperformed medical-specific models, and incorporating
EMPEC's training data significantly enhanced performance. Additionally, the
results on questions released after the models' training cutoff date were
consistent with overall performance trends, suggesting that the models'
performance on the test set can predict their effectiveness in addressing
unseen healthcare-related queries. The transition from traditional to
simplified Chinese characters had a negligible impact on model performance,
indicating robust linguistic versatility. Our study underscores the importance
of expanding benchmarks to cover a broader range of healthcare professions to
better assess the applicability of LLMs in real-world healthcare scenarios.

通过引入 “中文医务人员考试”（EMPEC），我们在传统中文领域提出了一个规模庞大的医疗知识基准，它由 157,803 个考试题目组成，涵盖 124 个科目和 20 个医疗专业，包括眼科医生和听力学家等被较少涉及的职业。我们的实验表明，尽管 GPT-4 等领先模型的准确率超过 75％，但它们在特殊领域和替代医学方面仍然存在困难。意外的是，通用领域的大型语言模型表现优于专门的医疗模型，并且融入 EMPEC 的训练数据显著提高了性能。此外，模型在训练截止日期后发布的问题上的结果与整体性能趋势一致，说明测试集上的模型表现可以预测其解决未见过的与医疗相关的查询的效果。传统汉字向简化汉字的过渡对模型性能影响微乎其微，表明其具有强大的语言适应性。我们的研究强调了扩大基准范围以涵盖更广泛的医疗职业的重要性，以更好地评估大型语言模型在实际医疗场景中的适用性。