It is intractable to evaluate the performance of Grammatical Error Correction
(GEC) systems since GEC is a highly subjective task. Designing an evaluation
metric that is as objective as possible is crucial to the development of GEC
task. Previous mainstream evaluation metrics, i.e., reference-based metrics,
introduce bias into the multi-reference evaluation because they extract edits
without considering the presence of multiple references. To overcome the
problem, we propose Chunk-LEvel Multi-reference Evaluation (CLEME) designed to
evaluate GEC systems in multi-reference settings. First, CLEME builds chunk
sequences with consistent boundaries for the source, the hypothesis and all the
references, thus eliminating the bias caused by inconsistent edit boundaries.
Then, based on the discovery that there exist boundaries between different
grammatical errors, we automatically determine the grammatical error boundaries
and compute F$_{0.5}$ scores in a novel way. Our proposed CLEME approach
consistently and substantially outperforms existing reference-based GEC metrics
on multiple reference sets in both corpus-level and sentence-level settings.
Extensive experiments and detailed analyses demonstrate the correctness of our
discovery and the effectiveness of our designed evaluation metric.

提出了一种 Chunk-Level Multi-reference Evaluation (CLEME) 方法，通过建立一致边界的块序列并自动确定语法错误边界，以评估多引用设置下的语法纠错系统的性能，CLEME 方法在语料库级别和句子级别设置中比现有基于参考的 GEC 度量表现更好。

CLEME: 语法错误修正的去偏置多参考评估

CLEME: Debiasing Multi-reference Evaluation for Grammatical Error  Correction

The aim of this paper is to mitigate the shortcomings of automatic evaluation
of open-domain dialog systems through multi-reference evaluation. Existing
metrics have been shown to correlate poorly with human judgement, particularly
in open-domain dialog. One alternative is to collect human annotations for
evaluation, which can be expensive and time consuming. To demonstrate the
effectiveness of multi-reference evaluation, we augment the test set of
DailyDialog with multiple references. A series of experiments show that the use
of multiple references results in improved correlation between several
automatic metrics and human judgement for both the quality and the diversity of
system output.

本文旨在通过多参考评估减少开放领域对话系统的自动评估的缺点，研究结果表明，使用多参考评估可以提高几种自动指标与人类判断之间的相关性，无论是对于系统输出的质量还是多样性。