Coherence is a linguistic term that refers to the relations between small
textual units (sentences, propositions), which make the text logically
consistent and meaningful to the reader. With the advances of generative
foundational models in NLP, there is a pressing need to automatically assess
the human-perceived coherence of automatically generated texts. Up until now,
little work has been done on explicitly assessing the coherence of generated
texts and analyzing the factors contributing to (in)coherence. Previous work on
the topic used other tasks, e.g., sentence reordering, as proxies of coherence,
rather than approaching coherence detection heads on. In this paper, we
introduce {\sc CoheSentia}, a novel benchmark of human-perceived coherence of
automatically generated texts. Our annotation protocol reflects two
perspectives; one is global, assigning a single coherence score, and the other
is incremental, scoring sentence by sentence. The incremental method produces
an (in)coherence score for each text fragment and also pinpoints reasons for
incoherence at that point. Our benchmark contains 500 automatically-generated
and human-annotated paragraphs, each annotated in both methods, by multiple
raters. Our analysis shows that the inter-annotator agreement in the
incremental mode is higher than in the holistic alternative, and our
experiments show that standard LMs fine-tuned for coherence detection show
varied performance on the different factors contributing to (in)coherence. All
in all, these models yield unsatisfactory performance, emphasizing the need for
developing more reliable methods for coherence assessment.

自然语言处理中自动生成文本的一致性评估和分析的新基准，旨在发展更可靠的一致性评估方法。

CoheSentia：生成文本中增量与整体一致性评估的新型基准

CoheSentia: A Novel Benchmark of Incremental versus Holistic Assessment  of Coherence in Generated Texts

Coherence is an essential property of well-written texts, that refers to the
way textual units relate to one another. In the era of generative AI, coherence
assessment is essential for many NLP tasks; summarization, generation,
long-form question-answering, and more. However, in NLP {coherence} is an
ill-defined notion, not having a formal definition or evaluation metrics, that
would allow for large-scale automatic and systematic coherence assessment. To
bridge this gap, in this work we employ the formal linguistic definition of
\citet{Reinhart:1980} of what makes a discourse coherent, consisting of three
conditions -- {\em cohesion, consistency} and {\em relevance} -- and formalize
these conditions as respective computational tasks. We hypothesize that (i) a
model trained on all of these tasks will learn the features required for
coherence detection, and that (ii) a joint model for all tasks will exceed the
performance of models trained on each task individually. On two benchmarks for
coherence scoring rated by humans, one containing 500 automatically-generated
short stories and another containing 4k real-world texts, our experiments
confirm that jointly training on the proposed tasks leads to better performance
on each task compared with task-specific models, and to better performance on
assessing coherence overall, compared with strong baselines. We conclude that
the formal and computational setup of coherence as proposed here provides a
solid foundation for advanced methods of large-scale automatic assessment of
coherence.

该论文提出了一种正式和计算的口头表达的一致性的设置和评估方法，该方法通过联合训练多种任务的模型，在评估一致性的各个任务上表现出更好的性能，为大规模自动评估一致性提供了坚实的基础。

自动一致性评估的新型计算和建模基础

A Novel Computational and Modeling Foundation for Automatic Coherence  Assessment

Automatic emotion categorization has been predominantly formulated as text
classification in which textual units are assigned to an emotion from a
predefined inventory, for instance following the fundamental emotion classes
proposed by Paul Ekman (fear, joy, anger, disgust, sadness, surprise) or Robert
Plutchik (adding trust, anticipation). This approach ignores existing
psychological theories to some degree, which provide explanations regarding the
perception of events. For instance, the description that somebody discovers a
snake is associated with fear, based on the appraisal as being an unpleasant
and non-controllable situation. This emotion reconstruction is even possible
without having access to explicit reports of a subjective feeling (for instance
expressing this with the words "I am afraid."). Automatic classification
approaches therefore need to learn properties of events as latent variables
(for instance that the uncertainty and the mental or physical effort associated
with the encounter of a snake leads to fear). With this paper, we propose to
make such interpretations of events explicit, following theories of cognitive
appraisal of events, and show their potential for emotion classification when
being encoded in classification models. Our results show that high quality
appraisal dimension assignments in event descriptions lead to an improvement in
the classification of discrete emotion categories. We make our corpus of
appraisal-annotated emotion-associated event descriptions publicly available.

本文旨在将心理学中事件的应急评估理论应用到情感分类模型中，实现对事件描述的应急评估变量的学习，证明这种方法能提高分类模型对离散情感分类的准确性，并发布了应急评估标记的情感关联事件描述语料库。