Summarizing comparative opinions about entities (e.g., hotels, phones) from a
set of source reviews, often referred to as contrastive summarization, can
considerably aid users in decision making. However, reliably measuring the
contrastiveness of the output summaries without relying on human evaluations
remains an open problem. Prior work has proposed token-overlap based metrics,
Distinctiveness Score, to measure contrast which does not take into account the
sensitivity to meaning-preserving lexical variations. In this work, we propose
an automated evaluation metric CASPR to better measure contrast between a pair
of summaries. Our metric is based on a simple and light-weight method that
leverages natural language inference (NLI) task to measure contrast by
segmenting reviews into single-claim sentences and carefully aggregating NLI
scores between them to come up with a summary-level score. We compare CASPR
with Distinctiveness Score and a simple yet powerful baseline based on
BERTScore. Our results on a prior dataset CoCoTRIP demonstrate that CASPR can
more reliably capture the contrastiveness of the summary pairs compared to the
baselines.

本文提出了一种基于自然语言推理（NLI）的自动评估指标 CASPR，用于更好地测量摘要对之间的对比度，并通过对比实验结果表明，CASPR 相比于其他基准方法能够更可靠地捕捉到摘要对之间的对比度。