Recent studies on automatic note generation have shown that doctors can save
significant amounts of time when using automatic clinical note generation
(Knoll et al., 2022). Summarization models have been used for this task to
generate clinical notes as summaries of doctor-patient conversations (Krishna
et al., 2021; Cai et al., 2022). However, assessing which model would best
serve clinicians in their daily practice is still a challenging task due to the
large set of possible correct summaries, and the potential limitations of
automatic evaluation metrics. In this paper, we study evaluation methods and
metrics for the automatic generation of clinical notes from medical
conversations. In particular, we propose new task-specific metrics and we
compare them to SOTA evaluation metrics in text summarization and generation,
including: (i) knowledge-graph embedding-based metrics, (ii) customized
model-based metrics, (iii) domain-adapted/fine-tuned metrics, and (iv) ensemble
metrics. To study the correlation between the automatic metrics and manual
judgments, we evaluate automatic notes/summaries by comparing the system and
reference facts and computing the factual correctness, and the hallucination
and omission rates for critical medical facts. This study relied on seven
datasets manually annotated by domain experts. Our experiments show that
automatic evaluation metrics can have substantially different behaviors on
different types of clinical notes datasets. However, the results highlight one
stable subset of metrics as the most correlated with human judgments with a
relevant aggregation of different evaluation criteria.

本文旨在研究自动生成临床笔记的评估方法和度量标准，提出了新的任务特定度量标准，并将其与现有文本摘要和生成的 SOTA 度量标准进行比较，研究分布和测量不同的自动摘要的正确性，并试图确定与人类判断相一致的最佳自动评估指标。