Neural language models have exhibited outstanding performance in a range of
downstream tasks. However, there is limited understanding regarding the extent
to which these models internalize syntactic knowledge, so that various datasets
have recently been constructed to facilitate syntactic evaluation of language
models across languages. In this paper, we introduce JCoLA (Japanese Corpus of
Linguistic Acceptability), which consists of 10,020 sentences annotated with
binary acceptability judgments. Specifically, those sentences are manually
extracted from linguistics textbooks, handbooks and journal articles, and split
into in-domain data (86 %; relatively simple acceptability judgments extracted
from textbooks and handbooks) and out-of-domain data (14 %; theoretically
significant acceptability judgments extracted from journal articles), the
latter of which is categorized by 12 linguistic phenomena. We then evaluate the
syntactic knowledge of 9 different types of Japanese language models on JCoLA.
The results demonstrated that several models could surpass human performance
for the in-domain data, while no models were able to exceed human performance
for the out-of-domain data. Error analyses by linguistic phenomena further
revealed that although neural language models are adept at handling local
syntactic dependencies like argument structure, their performance wanes when
confronted with long-distance syntactic dependencies like verbal agreement and
NPI licensing.

通过 JCoLA 数据集，对 9 种不同类型的日语语言模型进行了句法知识评估，结果表明有些模型在领域内数据上能超越人类表现，但在领域外数据上无法超越人类表现，并且通过对语言现象的错误分析发现，神经语言模型在处理像参数结构这样的局部语法依赖时表现出色，但在面对像动词一致和 NPI 授权这样的长距离语法依赖时表现出衰退。