LLMs have achieved significant performance progress in various NLP
applications. However, LLMs still struggle to meet the strict requirements for
accuracy and reliability in the medical field and face many challenges in
clinical applications. Existing clinical diagnostic evaluation benchmarks for
evaluating medical agents powered by LLMs have severe limitations. Firstly,
most existing medical evaluation benchmarks face the risk of data leakage or
contamination. Secondly, existing benchmarks often neglect the characteristics
of multiple departments and specializations in modern medical practice.
Thirdly, existing evaluation methods are limited to multiple-choice questions,
which do not align with the real-world diagnostic scenarios. Lastly, existing
evaluation methods lack comprehensive evaluations of end-to-end real clinical
scenarios. These limitations in benchmarks in turn obstruct advancements of
LLMs and agents for medicine. To address these limitations, we introduce
ClinicalLab, a comprehensive clinical diagnosis agent alignment suite.
ClinicalLab includes ClinicalBench, an end-to-end multi-departmental clinical
diagnostic evaluation benchmark for evaluating medical agents and LLMs.
ClinicalBench is based on real cases that cover 24 departments and 150
diseases. ClinicalLab also includes four novel metrics (ClinicalMetrics) for
evaluating the effectiveness of LLMs in clinical diagnostic tasks. We evaluate
17 LLMs and find that their performance varies significantly across different
departments. Based on these findings, in ClinicalLab, we propose ClinicalAgent,
an end-to-end clinical agent that aligns with real-world clinical diagnostic
practices. We systematically investigate the performance and applicable
scenarios of variants of ClinicalAgent on ClinicalBench. Our findings
demonstrate the importance of aligning with modern medical practices in
designing medical agents.

LLMs 在医学领域中面临精度和可靠性要求严苛的挑战，现有临床评估基准存在诸多限制，为解决这些限制，我们介绍了 ClinicalLab，它包括基于真实病例的 ClinicalBench 评估基准以及用于评估 LLMs 效果的四个新指标（ClinicalMetrics），我们还提出了与现实临床诊断实践相符的综合性临床代理（ClinicalAgent）。