Assessing the quality of summarizers poses significant challenges. In
response, we propose a novel task-oriented evaluation approach that assesses
summarizers based on their capacity to produce summaries that are useful for
downstream tasks, while preserving task outcomes. We theoretically establish a
direct relationship between the resulting error probability of these tasks and
the mutual information between source texts and generated summaries. We
introduce $\texttt{COSMIC}$ as a practical implementation of this metric,
demonstrating its strong correlation with human judgment-based metrics and its
effectiveness in predicting downstream task performance. Comparative analyses
against established metrics like $\texttt{BERTScore}$ and $\texttt{ROUGE}$
highlight the competitive performance of $\texttt{COSMIC}$.

我们提出了一种新颖的面向任务的评估方法，评估摘要生成器的质量，同时保持任务成果；我们在理论上建立了任务结果的错误概率与源文本和生成摘要之间的互信息之间的直接关系；我们引入了 COSMIC 作为这一度量的实际实现，证明其与人类判断基于度量的密切相关以及其在预测下游任务性能上的有效性；与 BERTScore 和 ROUGE 等已有度量的比较分析突显了 COSMIC 的竞争性能。

COSMIC: 任务无关摘要评估的互信息

$\texttt{COSMIC}$: Mutual Information for Task-Agnostic Summarization  Evaluation

The automatic scoring of summaries is important as it guides the development
of summarizers. Scoring is also complex, as it involves multiple aspects such
as fluency, grammar, and even textual entailment with the source text. However,
summary scoring has not been considered a machine learning task to study its
accuracy and robustness. In this study, we place automatic scoring in the
context of regression machine learning tasks and perform evasion attacks to
explore its robustness. Attack systems predict a non-summary string from each
input, and these non-summary strings achieve competitive scores with good
summarizers on the most popular metrics: ROUGE, METEOR, and BERTScore. Attack
systems also "outperform" state-of-the-art summarization methods on ROUGE-1 and
ROUGE-L, and score the second-highest on METEOR. Furthermore, a BERTScore
backdoor is observed: a simple trigger can score higher than any automatic
summarization method. The evasion attacks in this work indicate the low
robustness of current scoring systems at the system level. We hope that our
highlighting of these proposed attacks will facilitate the development of
summary scores.

本研究针对自动摘要的评分问题进行了机器学习和逃避攻击，发现当前自动评分系统的鲁棒性较低，需要进一步改进。