Long-form clinical summarization of hospital admissions has real-world
significance because of its potential to help both clinicians and patients. The
faithfulness of summaries is critical to their safe usage in clinical settings.
To better understand the limitations of abstractive systems, as well as the
suitability of existing evaluation metrics, we benchmark faithfulness metrics
against fine-grained human annotations for model-generated summaries of a
patient's Brief Hospital Course. We create a corpus of patient hospital
admissions and summaries for a cohort of HIV patients, each with complex
medical histories. Annotators are presented with summaries and source notes,
and asked to categorize manually highlighted summary elements (clinical
entities like conditions and medications as well as actions like "following
up") into one of three categories: ``Incorrect,'' ``Missing,'' and ``Not in
Notes.'' We meta-evaluate a broad set of proposed faithfulness metrics and,
across metrics, explore the importance of domain adaptation (e.g. the impact of
in-domain pre-training and metric fine-tuning), the use of source-summary
alignments, and the effects of distilling a single metric from an ensemble of
pre-existing metrics. Off-the-shelf metrics with no exposure to clinical text
correlate well yet overly rely on summary extractiveness. As a practical guide
to long-form clinical narrative summarization, we find that most metrics
correlate best to human judgments when provided with one summary sentence at a
time and a minimal set of relevant source context.

本研究基于 HIV 病人的病例史，对临床文本摘要的可信度指标进行评估，从医学专业人员和患者两方面来获得实际应用价值。评估表明，大多数指标在提供与源文本相关的最小信息上，可显著提高其与人类判断的相关性。