Natural language processing technology has rapidly improved automated
grammatical error correction tasks, and the community begins to explore
document-level revision as one of the next challenges. To go beyond
sentence-level automated grammatical error correction to NLP-based
document-level revision assistant, there are two major obstacles: (1) there are
few public corpora with document-level revisions being annotated by
professional editors, and (2) it is not feasible to elicit all possible
references and evaluate the quality of revision with such references because
there are infinite possibilities of revision. This paper tackles these
challenges. First, we introduce a new document-revision corpus, TETRA, where
professional editors revised academic papers sampled from the ACL anthology
which contain few trivial grammatical errors that enable us to focus more on
document- and paragraph-level edits such as coherence and consistency. Second,
we explore reference-less and interpretable methods for meta-evaluation that
can detect quality improvements by document revision. We show the uniqueness of
TETRA compared with existing document revision corpora and demonstrate that a
fine-tuned pre-trained language model can discriminate the quality of documents
after revision even when the difference is subtle. This promising result will
encourage the community to further explore automated document revision models
and metrics in future.

本文介绍了一个基于 TETRA 语料库的文档级修订助手。我们探索了无需参考评估和可解释的方法来评估文档修订的质量。实验结果表明，即使修订的差异微小，经过微调的预训练语言模型也能够区分文档修订后的质量，这为未来探索自动文档修订模型和评估指标奠定了基础。