Motivated by recent findings on the probabilistic modeling of acceptability
judgments, we propose syntactic log-odds ratio (SLOR), a normalized language
model score, as a metric for referenceless fluency evaluation of natural
language generation output at the sentence level. We further introduce WPSLOR,
a novel WordPiece-based version, which harnesses a more compact language model.
Even though word-overlap metrics like ROUGE are computed with the help of
hand-written references, our referenceless methods obtain a significantly
higher correlation with human fluency scores on a benchmark dataset of
compressed sentences. Finally, we present ROUGE-LM, a reference-based metric
which is a natural extension of WPSLOR to the case of available references. We
show that ROUGE-LM yields a significantly higher correlation with human
judgments than all baseline metrics, including WPSLOR on its own.

本文提出了基于正态化的语言模型得分的句子级无参考流畅度评估度量 SLOR，并介绍了一种更紧凑的语言模型 WPSLOR。本研究还引入了基于参考的度量 ROUGE-LM，该度量在可用参考下获得了更高的相关性。