With the growing interest in large language models, the need for evaluating
the quality of machine text compared to reference (typically human-generated)
text has become focal attention. Most recent works focus either on
task-specific evaluation metrics or study the properties of machine-generated
text captured by the existing metrics. In this work, we propose a new
evaluation scheme to model human judgments in 7 NLP tasks, based on the
fine-grained mismatches between a pair of texts. Inspired by the recent efforts
in several NLP tasks for fine-grained evaluation, we introduce a set of 13
mismatch error types such as spatial/geographic errors, entity errors, etc, to
guide the model for better prediction of human judgments. We propose a neural
framework for evaluating machine texts that uses these mismatch error types as
auxiliary tasks and re-purposes the existing single-number evaluation metrics
as additional scalar features, in addition to textual features extracted from
the machine and reference texts. Our experiments reveal key insights about the
existing metrics via the mismatch errors. We show that the mismatch errors
between the sentence pairs on the held-out datasets from 7 NLP tasks align well
with the human evaluation.

本文提出了一种新的评估模型人类的七种 NLP 任务的机器文本的方案，并通过副任务和现有单一数字评估指标对模型进行了改进。

MISMATCH: 机器生成文本的细粒度误差类型评估

MISMATCH: Fine-grained Evaluation of Machine-generated Text with  Mismatch Error Types

While neural language models can generate text with remarkable fluency and
coherence, controlling for factual correctness in generation remains an open
research question. This major discrepancy between the surface-level fluency and
the content-level correctness of neural generation has motivated a new line of
research that seeks automatic metrics for evaluating the factuality of machine
text. In this paper, we introduce GO FIGURE, a meta-evaluation framework for
evaluating factuality evaluation metrics. We propose five necessary and
intuitive conditions to evaluate factuality metrics on diagnostic factuality
data across three different summarization tasks. Our benchmark analysis on ten
factuality metrics reveals that our meta-evaluation framework provides a robust
and efficient evaluation that is extensible to multiple types of factual
consistency and standard generation metrics, including QA metrics. It also
reveals that while QA metrics generally improve over standard metrics that
measure factuality across domains, performance is highly dependent on the way
in which questions are generated.

本研究提出了一个基于事实准确性的机器文本自动生成质量评价框架 ——GO FIGURE，针对 10 种不同的事实准确性评价指标进行了评估，结果表明 QA 指标相较于标准指标具有更强的性能，但性能仍高度依赖问题的生成方式。