Coherence is a linguistic term that refers to the relations between small
textual units (sentences, propositions), which make the text logically
consistent and meaningful to the reader. With the advances of generative
foundational models in NLP, there is a pressing need to automatically assess
the human-perceived coherence of automatically generated texts. Up until now,
little work has been done on explicitly assessing the coherence of generated
texts and analyzing the factors contributing to (in)coherence. Previous work on
the topic used other tasks, e.g., sentence reordering, as proxies of coherence,
rather than approaching coherence detection heads on. In this paper, we
introduce {\sc CoheSentia}, a novel benchmark of human-perceived coherence of
automatically generated texts. Our annotation protocol reflects two
perspectives; one is global, assigning a single coherence score, and the other
is incremental, scoring sentence by sentence. The incremental method produces
an (in)coherence score for each text fragment and also pinpoints reasons for
incoherence at that point. Our benchmark contains 500 automatically-generated
and human-annotated paragraphs, each annotated in both methods, by multiple
raters. Our analysis shows that the inter-annotator agreement in the
incremental mode is higher than in the holistic alternative, and our
experiments show that standard LMs fine-tuned for coherence detection show
varied performance on the different factors contributing to (in)coherence. All
in all, these models yield unsatisfactory performance, emphasizing the need for
developing more reliable methods for coherence assessment.

自然语言处理中自动生成文本的一致性评估和分析的新基准，旨在发展更可靠的一致性评估方法。

CoheSentia：生成文本中增量与整体一致性评估的新型基准

CoheSentia: A Novel Benchmark of Incremental versus Holistic Assessment  of Coherence in Generated Texts

Recent dialogue coherence models use the coherence features designed for
monologue texts, e.g. nominal entities, to represent utterances and then
explicitly augment them with dialogue-relevant features, e.g., dialogue act
labels. It indicates two drawbacks, (a) semantics of utterances is limited to
entity mentions, and (b) the performance of coherence models strongly relies on
the quality of the input dialogue act labels. We address these issues by
introducing a novel approach to dialogue coherence assessment. We use dialogue
act prediction as an auxiliary task in a multi-task learning scenario to obtain
informative utterance representations for coherence assessment. Our approach
alleviates the need for explicit dialogue act labels during evaluation. The
results of our experiments show that our model substantially (more than 20
accuracy points) outperforms its strong competitors on the DailyDialogue
corpus, and performs on par with them on the SwitchBoard corpus for ranking
dialogues concerning their coherence.

本文提出了一种利用对话行为预测进行多任务学习的新方法，用于对话连贯性评估，并消除了对显式对话行为标签的需求，有效解决了以往研究领域中语义限制有限以及性能强烈依赖于输入对话行为标签等问题。

没有明确对话行为标签的对话连贯性评估

Dialogue Coherence Assessment Without Explicit Dialogue Act Labels

Coherence across multiple turns is a major challenge for state-of-the-art
dialogue models. Arguably the most successful approach to automatically
learning text coherence is the entity grid, which relies on modelling patterns
of distribution of entities across multiple sentences of a text. Originally
applied to the evaluation of automatic summaries and the news genre, among its
many extensions, this model has also been successfully used to assess dialogue
coherence. Nevertheless, both the original grid and its extensions do not model
intents, a crucial aspect that has been studied widely in the literature in
connection to dialogue structure. We propose to augment the original grid
document representation for dialogue with the intentional structure of the
conversation. Our models outperform the original grid representation on both
text discrimination and insertion, the two main standard tasks for coherence
assessment across three different dialogue datasets, confirming that intents
play a key role in modelling dialogue coherence.

在对话模型中，意图结构在话语的连贯性建模中起着关键作用。本文提出通过增强原始实体方格建模对话文档表示，实现对话中意图结构的建模，并在连贯性评估的两项标准任务方面对三个不同对话数据集的性能都优于原始实体方格表征，验证了意图在对话连贯性建模中的关键作用。