To understand what kinds of linguistic knowledge are encoded by pretrained
Chinese language models (LMs), we introduce the benchmark of Sino LINGuistics
(SLING), which consists of 38K minimal sentence pairs in Mandarin Chinese
grouped into 9 high-level linguistic phenomena. Each pair demonstrates the
acceptability contrast of a specific syntactic or semantic phenomenon (e.g.,
The keys are lost vs. The keys is lost), and an LM should assign lower
perplexity to the acceptable sentence. In contrast to the CLiMP dataset (Xiang
et al., 2021), which also contains Chinese minimal pairs and was created by
translating the vocabulary of the English BLiMP dataset, the minimal pairs in
SLING are derived primarily by applying syntactic and lexical transformations
to naturally-occurring, linguist-annotated sentences from the Chinese Treebank
9.0, thus addressing severe issues in CLiMP's data generation process. We test
18 publicly available pretrained monolingual (e.g., BERT-base-zh, CPM) and
multi-lingual (e.g., mT5, XLM) language models on SLING. Our experiments show
that the average accuracy for LMs is far below human performance (69.7% vs.
97.1%), while BERT-base-zh achieves the highest accuracy (84.8%) of all tested
LMs, even much larger ones. Additionally, we find that most LMs have a strong
gender and number (singular/plural) bias, and they perform better on local
phenomena than hierarchical ones.

通过引入基准 SLING，作者测试了 18 个预训练的母语和多语言模型，研究它们对汉语句子的理解效果以及现象级别、社会性别和单 / 复数等方面的影响。作者发现预训练语言模型在 SLING 上的平均准确率远低于人类表现，并存在很强的个别现象偏见。