Though state-of-the-art sentence representation models can perform tasks
requiring significant knowledge of grammar, it is an open question how best to
evaluate their grammatical knowledge. We explore five experimental methods
inspired by prior work evaluating pretrained sentence representation models. We
use a single linguistic phenomenon, negative polarity item (NPI) licensing in
English, as a case study for our experiments. NPIs like "any" are grammatical
only if they appear in a licensing environment like negation ("Sue doesn't have
any cats" vs. "Sue has any cats"). This phenomenon is challenging because of
the variety of NPI licensing environments that exist. We introduce an
artificially generated dataset that manipulates key features of NPI licensing
for the experiments. We find that BERT has significant knowledge of these
features, but its success varies widely across different experimental methods.
We conclude that a variety of methods is necessary to reveal all relevant
aspects of a model's grammatical knowledge in a given domain.

本文探讨评估预训练句子表示模型的语法知识的五种实验方法，并以英语中的负极性项（NPI）许可作为案例研究。作者发现 BERT 对关键特征有较高的了解，但其成功程度在不同的实验方法中存在差异。作者认为在其给定领域内揭示模型语法知识的各个相关方面需要多种方法。