The goal of a summary is to concisely state the most important information in a document. With this principle in mind, we introduce new reference-free summary evaluation metrics that use a pretrained language model to estimate the information shared between a document and its summary. These metrics are a modern take on the Shannon Game, a method for summary quality scoring proposed decades ago, where we replace human annotators with language models. We also view these metrics as an extension of BLANC, a recently proposed approach to summary quality measurement based on the performance of a language model with and without the help of a summary. Using GPT-2, we empirically verify that the introduced metrics correlate with human judgement based on coverage, overall quality, and five summary dimensions.

本篇论文提出了一种新型无需参考文献的摘要评价指标，利用预训练语言模型来评估文档和其摘要之间共享的信息内容，此指标是之前提出的摘要质量评分方法Shannon Game的现代演化版本，并通过实验证明，基于Transformer的语言模型得到的指标与人类判断摘要质量的相关性达到了最先进的水平，可用于衡量摘要的连贯性、相关性、一致性和流畅性。

使用语言模型玩Shannon游戏: 一种无需人类参与的文章摘要评估方法