Clinical problem-solving requires processing of semantic medical knowledge
such as illness scripts and numerical medical knowledge of diagnostic tests for
evidence-based decision-making. As large language models (LLMs) show promising
results in many aspects of language-based clinical practice, their ability to
generate non-language evidence-based answers to clinical questions is
inherently limited by tokenization. Therefore, we evaluated LLMs' performance
on two question types: numeric (correlating findings) and semantic
(differentiating entities) while examining differences within and between LLMs
in medical aspects and comparing their performance to humans. To generate
straightforward multi-choice questions and answers (QAs) based on
evidence-based medicine (EBM), we used a comprehensive medical knowledge graph
(encompassed data from more than 50,00 peer-reviewed articles) and created the
"EBMQA". EBMQA contains 105,000 QAs labeled with medical and non-medical topics
and classified into numerical or semantic questions. We benchmarked this
dataset using more than 24,500 QAs on two state-of-the-art LLMs: Chat-GPT4 and
Claude3-Opus. We evaluated the LLMs accuracy on semantic and numerical question
types and according to sub-labeled topics. For validation, six medical experts
were tested on 100 numerical EBMQA questions. We found that both LLMs excelled
more in semantic than numerical QAs, with Claude3 surpassing GPT4 in numerical
QAs. However, both LLMs showed inter and intra gaps in different medical
aspects and remained inferior to humans. Thus, their medical advice should be
addressed carefully.

临床问题解决需要处理语义医学知识，如疾病描述和诊断测试的数值医学知识进行循证决策。因此，我们评估了大型语言模型（LLMs）在数值和语义问题类型上的表现，并将其与人类进行比较。研究发现，LLMs 在语义问题上的表现优于数值问题，在不同的医学方面存在差距，仍然不及人类，因此应该谨慎对待它们的医疗建议。