We conduct a thorough study to diagnose the behaviors of pre-trained language
encoders (ELMo, BERT, and RoBERTa) when confronted with natural grammatical
errors. Specifically, we collect real grammatical errors from non-native
speakers and conduct adversarial attacks to simulate these errors on clean text
data. We use this approach to facilitate debugging models on downstream
applications. Results confirm that the performance of all tested models is
affected but the degree of impact varies. To interpret model behaviors, we
further design a linguistic acceptability task to reveal their abilities in
identifying ungrammatical sentences and the position of errors. We find that
fixed contextual encoders with a simple classifier trained on the prediction of
sentence correctness are able to locate error positions. We also design a cloze
test for BERT and discover that BERT captures the interaction between errors
and specific tokens in context. Our results shed light on understanding the
robustness and behaviors of language encoders against grammatical errors.

本文旨在研究预训练语言编码器（ELMo，BERT 和 RoBERTa）在面对自然语法错误时的表现，通过采集真实语法错误和进行对抗性攻击来模拟这些错误对干净文本数据的影响。结果证实，所有测试模型的性能都受到了影响，但影响程度有所不同。此外，我们设计了一个语言接受度任务来揭示它们在识别不符合语法的句子和错误位置方面的能力。本文的结果有助于理解语言编码器对语法错误的鲁棒性和行为。