With the ever-increasing complexity of neural language models, practitioners
have turned to methods for understanding the predictions of these models. One
of the most well-adopted approaches for model interpretability is feature-based
interpretability, i.e., ranking the features in terms of their impact on model
predictions. Several prior studies have focused on assessing the fidelity of
feature-based interpretability methods, i.e., measuring the impact of dropping
the top-ranked features on the model output. However, relatively little work
has been conducted on quantifying the robustness of interpretations. In this
work, we assess the robustness of interpretations of neural text classifiers,
specifically, those based on pretrained Transformer encoders, using two
randomization tests. The first compares the interpretations of two models that
are identical except for their initializations. The second measures whether the
interpretations differ between a model with trained parameters and a model with
random parameters. Both tests show surprising deviations from expected
behavior, raising questions about the extent of insights that practitioners may
draw from interpretations.

本研究通过两种随机化测试，评估预训练 Transformer 编码器的神经文本分类器的可解释性解释的鲁棒性，并发现出乎意料的偏差，从而对实践者从解释中获得的深入的见解提出疑问。

神经文本分类器的鲁棒可解释性缺失

On the Lack of Robust Interpretability of Neural Text Classifiers

Statistical significance testing of differences in values of metrics like
recall, precision and balanced F-score is a necessary part of empirical natural
language processing. Unfortunately, we find in a set of experiments that many
commonly used tests often underestimate the significance and so are less likely
to detect differences that exist between different techniques. This
underestimation comes from an independence assumption that is often violated.
We point out some useful tests that do not make this assumption, including
computationally-intensive randomization tests.

该研究探讨了在自然语言处理实证研究中，常用统计方法在独立性假设违反情况下，低估了统计差异的问题，并提出了一些不需要独立性假设的有效测试方法。