Is it possible to build a general and automatic natural language generation
(NLG) evaluation metric? Existing learned metrics either perform
unsatisfactorily or are restricted to tasks where large human rating data is
already available. We introduce SESCORE, a model-based metric that is highly
correlated with human judgements without requiring human annotation, by
utilizing a novel, iterative error synthesis and severity scoring pipeline.
This pipeline applies a series of plausible errors to raw text and assigns
severity labels by simulating human judgements with entailment. We evaluate
SESCORE against existing metrics by comparing how their scores correlate with
human ratings. SESCORE outperforms all prior unsupervised metrics on multiple
diverse NLG tasks including machine translation, image captioning, and WebNLG
text generation. For WMT 20/21 En-De and Zh-En, SESCORE improve the average
Kendall correlation with human judgement from 0.154 to 0.195. SESCORE even
achieves comparable performance to the best supervised metric COMET, despite
receiving no human-annotated training data.

本文提出利用迭代的错误合成与强度评分的新颖流程构建高度相关于人类判断的基于模型的指标 SESCORE，该指标无需人工注释，在多个不同的自然语言生成任务中优于所有现有的无监督指标，并且即使没有获得任何人工标注训练数据，SESCORE 也实现了与最佳监督指标 COMET 相当的性能。