Coherence is a linguistic term that refers to the relations between small
textual units (sentences, propositions), which make the text logically
consistent and meaningful to the reader. With the advances of generative
foundational models in NLP, there is a pressing need to automatically assess
the human-perceived coherence of automatically generated texts. Up until now,
little work has been done on explicitly assessing the coherence of generated
texts and analyzing the factors contributing to (in)coherence. Previous work on
the topic used other tasks, e.g., sentence reordering, as proxies of coherence,
rather than approaching coherence detection heads on. In this paper, we
introduce {\sc CoheSentia}, a novel benchmark of human-perceived coherence of
automatically generated texts. Our annotation protocol reflects two
perspectives; one is global, assigning a single coherence score, and the other
is incremental, scoring sentence by sentence. The incremental method produces
an (in)coherence score for each text fragment and also pinpoints reasons for
incoherence at that point. Our benchmark contains 500 automatically-generated
and human-annotated paragraphs, each annotated in both methods, by multiple
raters. Our analysis shows that the inter-annotator agreement in the
incremental mode is higher than in the holistic alternative, and our
experiments show that standard LMs fine-tuned for coherence detection show
varied performance on the different factors contributing to (in)coherence. All
in all, these models yield unsatisfactory performance, emphasizing the need for
developing more reliable methods for coherence assessment.

自然语言处理中自动生成文本的一致性评估和分析的新基准，旨在发展更可靠的一致性评估方法。