Large language models (LLMs) have performed remarkably well in various
natural language processing tasks by benchmarking, including in the Western
medical domain. However, the professional evaluation benchmarks for LLMs have
yet to be covered in the traditional Chinese medicine(TCM) domain, which has a
profound history and vast influence. To address this research gap, we introduce
TCM-Bench, an comprehensive benchmark for evaluating LLM performance in TCM. It
comprises the TCM-ED dataset, consisting of 5,473 questions sourced from the
TCM Licensing Exam (TCMLE), including 1,300 questions with authoritative
analysis. It covers the core components of TCMLE, including TCM basis and
clinical practice. To evaluate LLMs beyond accuracy of question answering, we
propose TCMScore, a metric tailored for evaluating the quality of answers
generated by LLMs for TCM related questions. It comprehensively considers the
consistency of TCM semantics and knowledge. After conducting comprehensive
experimental analyses from diverse perspectives, we can obtain the following
findings: (1) The unsatisfactory performance of LLMs on this benchmark
underscores their significant room for improvement in TCM. (2) Introducing
domain knowledge can enhance LLMs' performance. However, for in-domain models
like ZhongJing-TCM, the quality of generated analysis text has decreased, and
we hypothesize that their fine-tuning process affects the basic LLM
capabilities. (3) Traditional metrics for text generation quality like Rouge
and BertScore are susceptible to text length and surface semantic ambiguity,
while domain-specific metrics such as TCMScore can further supplement and
explain their evaluation results. These findings highlight the capabilities and
limitations of LLMs in the TCM and aim to provide a more profound assistance to
medical research.

介绍了 TCM-Bench 综合评估 TCM 中大型语言模型性能的基准，提出了 TCMScore 作为评估 TCM 问题回答质量的指标，并从不同角度进行了综合实验分析，突出了大型语言模型在 TCM 领域的能力和局限性。