Minimal pairs are a well-established approach to evaluating the grammatical
knowledge of language models. However, existing resources for minimal pairs
address a limited number of languages and lack diversity of language-specific
grammatical phenomena. This paper introduces the Russian Benchmark of
Linguistic Minimal Pairs (RuBLiMP), which includes 45k pairs of sentences that
differ in grammaticality and isolate a morphological, syntactic, or semantic
phenomenon. In contrast to existing benchmarks of linguistic minimal pairs,
RuBLiMP is created by applying linguistic perturbations to automatically
annotated sentences from open text corpora and carefully curating test data. We
describe the data collection protocol and present the results of evaluating 25
language models in various scenarios. We find that the widely used language
models for Russian are sensitive to morphological and agreement-oriented
contrasts but fall behind humans on phenomena requiring understanding of
structural relations, negation, transitivity, and tense. RuBLiMP, the codebase,
and other materials are publicly available.

这篇论文介绍了俄语语言最小对对比基准（RuBLiMP），它包含 45k 对的句子，这些句子在语法、形态、句法或语义现象上存在差异，与现有的语言最小对对比基准相比，RuBLiMP 利用了来自开放文本语料库的自动注释句子并仔细筛选测试数据。作者描述了数据收集协议，并展示了对 25 种语言模型在不同场景下的评估结果。研究发现，对于需要理解结构关系、否定、及时态的现象，俄语广泛使用的语言模型在形态和一致性方面较为敏感，但在这些现象上与人类相比存在一定差距。RuBLiMP、代码和其他材料可公开获取。

俄文语言最小对的评估基准：RuBLiMP

RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs

Though state-of-the-art sentence representation models can perform tasks
requiring significant knowledge of grammar, it is an open question how best to
evaluate their grammatical knowledge. We explore five experimental methods
inspired by prior work evaluating pretrained sentence representation models. We
use a single linguistic phenomenon, negative polarity item (NPI) licensing in
English, as a case study for our experiments. NPIs like "any" are grammatical
only if they appear in a licensing environment like negation ("Sue doesn't have
any cats" vs. "Sue has any cats"). This phenomenon is challenging because of
the variety of NPI licensing environments that exist. We introduce an
artificially generated dataset that manipulates key features of NPI licensing
for the experiments. We find that BERT has significant knowledge of these
features, but its success varies widely across different experimental methods.
We conclude that a variety of methods is necessary to reveal all relevant
aspects of a model's grammatical knowledge in a given domain.

本文探讨评估预训练句子表示模型的语法知识的五种实验方法，并以英语中的负极性项（NPI）许可作为案例研究。作者发现 BERT 对关键特征有较高的了解，但其成功程度在不同的实验方法中存在差异。作者认为在其给定领域内揭示模型语法知识的各个相关方面需要多种方法。