In recent years, machine learning models have rapidly become better at generating clinical consultation notes; yet, there is little work on how to properly evaluate the generated consultation notes to understand the impact they may have on both the clinician using them and the patient's clinical safety. To address this we present an extensive human evaluation study of consultation notes where 5 clinicians (i) listen to 57 mock consultations, (ii) write their own notes, (iii) post-edit a number of automatically generated notes, and (iv) extract all the errors, both quantitative and qualitative. We then carry out a correlation study with 18 automatic quality metrics and the human judgements. We find that a simple, character-based Levenshtein distance metric performs on par if not better than common model-based metrics like BertScore. All our findings and annotations are open-sourced.

研究机器学习模型如何生成临床咨询笔记，并提出了一种人工评估方法，以理解它们对医生和患者临床安全性的影响，发现基于字符的Levenshtein距离指标的性能与BertScore等基于模型的指标相当，甚至更好。

会诊记录生成中自动度量的人工评估和相关性