Large language models (LLMs) have achieved significant success in interacting
with human. However, recent studies have revealed that these models often
suffer from hallucinations, leading to overly confident but incorrect
judgments. This limits their application in the medical domain, where tasks
require the utmost accuracy. This paper introduces an automated evaluation
framework that assesses the practical capabilities of LLMs as virtual doctors
during multi-turn consultations. Consultation tasks are designed to require
LLMs to be aware of what they do not know, to inquire about missing medical
information from patients, and to ultimately make diagnoses. To evaluate the
performance of LLMs for these tasks, a benchmark is proposed by reformulating
medical multiple-choice questions from the United States Medical Licensing
Examinations (USMLE), and comprehensive evaluation metrics are developed and
evaluated on three constructed test sets. A medical consultation training set
is further constructed to improve the consultation ability of LLMs. The results
of the experiments show that fine-tuning with the training set can alleviate
hallucinations and improve LLMs' performance on the proposed benchmark.
Extensive experiments and ablation studies are conducted to validate the
effectiveness and robustness of the proposed framework.

该研究介绍了一种自动评估框架，评估了大型语言模型作为虚拟医生在多轮咨询中的实际能力，通过重构来自美国医疗执照考试的医学多项选择题，提出了一个基准测试集，并开发了综合评估指标。研究结果表明，使用培训集对大型语言模型进行微调，可以减轻幻觉并提高其在所提出基准上的性能。