Text summarization models are often trained to produce summaries that meet
human quality requirements. However, the existing evaluation metrics for
summary text are only rough proxies for summary quality, suffering from low
correlation with human scoring and inhibition of summary diversity. To solve
these problems, we propose SummScore, a comprehensive metric for summary
quality evaluation based on CrossEncoder. Firstly, by adopting the
original-summary measurement mode and comparing the semantics of the original
text, SummScore gets rid of the inhibition of summary diversity. With the help
of the text-matching pre-training Cross-Encoder, SummScore can effectively
capture the subtle differences between the semantics of summaries. Secondly, to
improve the comprehensiveness and interpretability, SummScore consists of four
fine-grained submodels, which measure Coherence, Consistency, Fluency, and
Relevance separately. We use semi-supervised multi-rounds of training to
improve the performance of our model on extremely limited annotated data.
Extensive experiments show that SummScore significantly outperforms existing
evaluation metrics in the above four dimensions in correlation with human
scoring. We also provide the quality evaluation results of SummScore on 16
mainstream summarization models for later research.

提出了一种综合的衡量文本摘要质量的度量标准 SummScore，使用 CrossEncoder 作为基础模型，并采用四个精细化的子模型来分别衡量连贯性、一致性、流畅度和相关性，并在极度受限的标注数据上进行半监督多轮训练，显著优于现有的评估指标。