It is known that a deep neural network model pre-trained with large-scale
data greatly improves the accuracy of various tasks, especially when there are
resource constraints. However, the information needed to solve a given task can
vary, and simply using the output of the final layer is not necessarily
sufficient. Moreover, to our knowledge, exploiting large language
representation models to detect grammatical errors has not yet been studied. In
this work, we investigate the effect of utilizing information not only from the
final layer but also from intermediate layers of a pre-trained language
representation model to detect grammatical errors. We propose a multi-head
multi-layer attention model that determines the appropriate layers in
Bidirectional Encoder Representation from Transformers (BERT). The proposed
method achieved the best scores on three datasets for grammatical error
detection tasks, outperforming the current state-of-the-art method by 6.0
points on FCE, 8.2 points on CoNLL14, and 12.2 points on JFLEG in terms of
F_0.5. We also demonstrate that by using multi-head multi-layer attention, our
model can exploit a broader range of information for each token in a sentence
than a model that uses only the final layer's information.

利用经过预训练的语言表示模型的中间和最终层提取句子中的语法错误特征，通过多头多层的注意力模型，使用 Bidirectional Encoder Representation from Transformers (BERT) ，在三个语法错误数据集上实现更好的 F_0.5 分数，比当前最先进的方法分别高出 6.0（FCE），8.2（CoNLL14）和 12.2（JFLEG）个百分点，并证明利用多头多层注意力，我们的模型可以利用句子中每个记号的更广泛信息。