Language models (LMs) are capable of acquiring elements of human-like syntactic knowledge. Targeted syntactic evaluation tests have been employed to measure how well they form generalizations about syntactic phenomena in high-resource languages such as English. However, we still lack a thorough understanding of LMs' capacity for syntactic generalizations in low-resource languages, which are responsible for much of the diversity of syntactic patterns worldwide. In this study, we develop targeted syntactic evaluation tests for three low-resource languages (Basque, Hindi, and Swahili) and use them to evaluate five families of open-access multilingual Transformer LMs. We find that some syntactic tasks prove relatively easy for LMs while others (agreement in sentences containing indirect objects in Basque, agreement across a prepositional phrase in Swahili) are challenging. We additionally uncover issues with publicly available Transformers, including a bias toward the habitual aspect in Hindi in multilingual BERT and underperformance compared to similar-sized models in XGLM-4.5B.

本研究解决了低资源语言中语言模型对句法概括能力理解不足的问题，通过针对性句法评估测试，对巴斯克语、印地语和斯瓦希里语进行评估。研究发现，模型在某些句法任务上表现良好，而在其他任务（如巴斯克语中的间接宾语一致性）上则面临挑战，并揭示了现有转换模型的偏差和性能不足问题。

多语种语言模型中句法知识的受控评估