As large and powerful neural language models are developed, researchers have
been increasingly interested in developing diagnostic tools to probe them.
There are many papers with conclusions of the form "observation X is found in
model Y", using their own datasets with varying sizes. Larger probing datasets
bring more reliability, but are also expensive to collect. There is yet to be a
quantitative method for estimating reasonable probing dataset sizes. We tackle
this omission in the context of comparing two probing configurations: after we
have collected a small dataset from a pilot study, how many additional data
samples are sufficient to distinguish two different configurations? We present
a novel method to estimate the required number of data samples in such
experiments and, across several case studies, we verify that our estimations
have sufficient statistical power. Our framework helps to systematically
construct probing datasets to diagnose neural NLP models.

本研究探索寻找一种数量化方法，以估算合理的探测数据集大小，旨在构建一个系统性的诊断神经自然语言处理 (NLP) 模型探测数据集的框架。通过几个案例研究，我们验证了我们的估算具有足够的统计功效。

关于探测的数据需求

On the data requirements of probing

Recent research has adopted a new experimental field centered around the
concept of text perturbations which has revealed that shuffled word order has
little to no impact on the downstream performance of Transformer-based language
models across many NLP tasks. These findings contradict the common
understanding of how the models encode hierarchical and structural information
and even question if the word order is modeled with position embeddings. To
this end, this paper proposes nine probing datasets organized by the type of
\emph{controllable} text perturbation for three Indo-European languages with a
varying degree of word order flexibility: English, Swedish and Russian. Based
on the probing analysis of the M-BERT and M-BART models, we report that the
syntactic sensitivity depends on the language and model pre-training
objectives. We also find that the sensitivity grows across layers together with
the increase of the perturbation granularity. Last but not least, we show that
the models barely use the positional information to induce syntactic trees from
their intermediate self-attention and contextualized representations.

探究了文本扰动对 Transformer-based 语言模型的影响，发现换序对模型下游任务性能影响不大，且证实模型对位置编码的使用。研究使用三种印欧语言建立九个 probing 数据集，发现句法敏感度取决于语言和预训练目标，在层数和扰动粒度上敏感度增加且模型几乎不用位置信息产生句法树。