Recently, there has been a growing interest in designing text generation
systems from a discourse coherence perspective, e.g., modeling the
interdependence between sentences. Still, recent BERT-based evaluation metrics
are weak in recognizing coherence, and thus are not reliable in a way to spot
the discourse-level improvements of those text generation systems. In this
work, we introduce DiscoScore, a parametrized discourse metric, which uses BERT
to model discourse coherence from different perspectives, driven by Centering
theory. Our experiments encompass 16 non-discourse and discourse metrics,
including DiscoScore and popular coherence models, evaluated on summarization
and document-level machine translation (MT). We find that (i) the majority of
BERT-based metrics correlate much worse with human rated coherence than early
discourse metrics, invented a decade ago; (ii) the recent state-of-the-art
BARTScore is weak when operated at system level -- which is particularly
problematic as systems are typically compared in this manner. DiscoScore, in
contrast, achieves strong system-level correlation with human ratings, not only
in coherence but also in factual consistency and other aspects, and surpasses
BARTScore by over 10 correlation points on average. Further, aiming to
understand DiscoScore, we provide justifications to the importance of discourse
coherence for evaluation metrics, and explain the superiority of one variant
over another. Our code is available at
https://github.com/AIPHES/DiscoScore.

介绍了一种基于 Centering 理论和 BERT 的参数化的 Discourse 评估度量，DiscoScore，在结构连贯性、事实一致性等方面优于当前流行的评估度量 BARTScore。