A range of studies have concluded that neural word prediction models can
distinguish grammatical from ungrammatical sentences with high accuracy.
However, these studies are based primarily on monolingual evidence from
English. To investigate how these models' ability to learn syntax varies by
language, we introduce CLAMS (Cross-Linguistic Assessment of Models on Syntax),
a syntactic evaluation suite for monolingual and multilingual models. CLAMS
includes subject-verb agreement challenge sets for English, French, German,
Hebrew and Russian, generated from grammars we develop. We use CLAMS to
evaluate LSTM language models as well as monolingual and multilingual BERT.
Across languages, monolingual LSTMs achieved high accuracy on dependencies
without attractors, and generally poor accuracy on agreement across object
relative clauses. On other constructions, agreement accuracy was generally
higher in languages with richer morphology. Multilingual models generally
underperformed monolingual models. Multilingual BERT showed high syntactic
accuracy on English, but noticeable deficiencies in other languages.

通过多语言的句法评估套件 CLAMS，调查神经词语预测模型在各种语言中学习语法的能力，结果表明单语 LSTMs 在句子成分及吸引子存在时达到了很高的精度，而在其他方面的一致性准确率较低，多语言模型通常表现不如单语模型。