The goal of automated summarization techniques (Paice, 1990; Kupiec et al,
1995) is to condense text by focusing on the most critical information.
Generative large language models (LLMs) have shown to be robust summarizers,
yet traditional metrics struggle to capture resulting performance (Goyal et al,
2022) in more powerful LLMs. In safety-critical domains such as medicine, more
rigorous evaluation is required, especially given the potential for LLMs to
omit important information in the resulting summary. We propose MED-OMIT, a new
omission benchmark for medical summarization. Given a doctor-patient
conversation and a generated summary, MED-OMIT categorizes the chat into a set
of facts and identifies which are omitted from the summary. We further propose
to determine fact importance by simulating the impact of each fact on a
downstream clinical task: differential diagnosis (DDx) generation. MED-OMIT
leverages LLM prompt-based approaches which categorize the importance of facts
and cluster them as supporting or negating evidence to the diagnosis. We
evaluate MED-OMIT on a publicly-released dataset of patient-doctor
conversations and find that MED-OMIT captures omissions better than alternative
metrics.

我们提出了一种针对医学摘要的新的遗漏评估基准 MED-OMIT，通过模拟每个事实对下游临床任务（鉴别诊断生成）的影响，将医生 - 患者对话和生成的摘要分类为一组事实并识别其在摘要中的遗漏。我们使用基于 LLM 提示的方法对事实的重要性进行排序并将其聚类为支持或否认诊断的证据。通过在公开数据集上评估 MED-OMIT，我们发现 MED-OMIT 比其他度量方法更好地捕捉到了遗漏。