There is an increasing interest in developing LLMs for medical diagnosis to
improve diagnosis efficiency. Despite their alluring technological potential,
there is no unified and comprehensive evaluation criterion, leading to the
inability to evaluate the quality and potential risks of medical LLMs, further
hindering the application of LLMs in medical treatment scenarios. Besides,
current evaluations heavily rely on labor-intensive interactions with LLMs to
obtain diagnostic dialogues and human evaluation on the quality of diagnosis
dialogue. To tackle the lack of unified and comprehensive evaluation criterion,
we first initially establish an evaluation criterion, termed LLM-specific
Mini-CEX to assess the diagnostic capabilities of LLMs effectively, based on
original Mini-CEX. To address the labor-intensive interaction problem, we
develop a patient simulator to engage in automatic conversations with LLMs, and
utilize ChatGPT for evaluating diagnosis dialogues automatically. Experimental
results show that the LLM-specific Mini-CEX is adequate and necessary to
evaluate medical diagnosis dialogue. Besides, ChatGPT can replace manual
evaluation on the metrics of humanistic qualities and provides reproducible and
automated comparisons between different LLMs.

为了提高诊断效率，增强医疗诊断的技术潜力，我们建立了一种评估标准 (LLM-specific Mini-CEX)，通过自动对话评估与 ChatGPT 的交互并替代人工评估，可以有效评估医疗诊断对话和比较不同 LLMs 之间的质量。