This study explores the necessity of performing cross-corpora evaluation for
grammatical error correction (GEC) models. GEC models have been previously
evaluated based on a single commonly applied corpus: the CoNLL-2014 benchmark.
However, the evaluation remains incomplete because the task difficulty varies
depending on the test corpus and conditions such as the proficiency levels of
the writers and essay topics. To overcome this limitation, we evaluate the
performance of several GEC models, including NMT-based (LSTM, CNN, and
transformer) and an SMT-based model, against various learner corpora
(CoNLL-2013, CoNLL-2014, FCE, JFLEG, ICNALE, and KJ). Evaluation results reveal
that the models' rankings considerably vary depending on the corpus, indicating
that single-corpus evaluation is insufficient for GEC models.

本文研究探讨语法纠错（GEC）模型跨语料库评估的必要性。我们评估了多个 GEC 模型（包括 NMT（LSTM、CNN 和 Transformer）和一个 SMT 模型），并发现单一语料库的评估不足以完整评估 GEC 模型的表现。

跨语料库评估和分析语法纠错模型 —— 单一语料库评估是否足够？

Cross-Corpora Evaluation and Analysis of Grammatical Error Correction  Models --- Is Single-Corpus Evaluation Enough?

Code corpora, as observed in large software systems, are now known to be far
more repetitive and predictable than natural language corpora. But why? Does
the difference simply arise from the syntactic limitations of programming
languages? Or does it arise from the differences in authoring decisions made by
the writers of these natural and programming language texts? We conjecture that
the differences are not entirely due to syntax, but also from the fact that
reading and writing code is un-natural for humans, and requires substantial
mental effort; so, people prefer to write code in ways that are familiar to
both reader and writer. To support this argument, we present results from two
sets of studies: 1) a first set aimed at attenuating the effects of syntax, and
2) a second, aimed at measuring repetitiveness of text written in other
settings (e.g. second language, technical/specialized jargon), which are also
effortful to write. We find find that this repetition in source code is not
entirely the result of grammar constraints, and thus some repetition must
result from human choice. While the evidence we find of similar repetitive
behavior in technical and learner corpora does not conclusively show that such
language is used by humans to mitigate difficulty, it is consistent with that
theory.

通过两组研究结果，得出代码重复性的原因不仅仅受语法限制，还受到了读写代码对人类认知负担的影响，人们更倾向于用熟悉的方式来书写代码；另外，技术和学习语言中的重复行为与减轻难度的假设相一致。