We present a dataset for evaluating the grammaticality of the predictions of
a language model. We automatically construct a large number of minimally
different pairs of English sentences, each consisting of a grammatical and an
ungrammatical sentence. The sentence pairs represent different variations of
structure-sensitive phenomena: subject-verb agreement, reflexive anaphora and
negative polarity items. We expect a language model to assign a higher
probability to the grammatical sentence than the ungrammatical one. In an
experiment using this data set, an LSTM language model performed poorly on many
of the constructions. Multi-task training with a syntactic objective (CCG
supertagging) improved the LSTM's accuracy, but a large gap remained between
its performance and the accuracy of human participants recruited online. This
suggests that there is considerable room for improvement over LSTMs in
capturing syntax in a language model.

该研究提出了使用一个语言模型预测的句子的语法正确性来评估该模型的数据集。实验结果表明，在众多句子变体中，LSTM 语言模型的准确性较差，而使用句法目标的多任务训练可以提高模型的准确性，但模型仍存在较大的进步空间。