Pre-trained language models (PLMs) like BERT are being used for almost all
language-related tasks, but interpreting their behavior still remains a
significant challenge and many important questions remain largely unanswered.
In this work, we re-purpose a sentence editing dataset, where faithful
high-quality human rationales can be automatically extracted and compared with
extracted model rationales, as a new testbed for interpretability. This enables
us to conduct a systematic investigation on an array of questions regarding
PLMs' interpretability, including the role of pre-training procedure,
comparison of rationale extraction methods, and different layers in the PLM.
The investigation generates new insights, for example, contrary to the common
understanding, we find that attention weights correlate well with human
rationales and work better than gradient-based saliency in extracting model
rationales. Both the dataset and code are available at
this https URL to
facilitate future interpretability research.

本文旨在将一个句子编辑数据集复用成为解释性测试场，系统研究预训练语言模型的可解释性及在该场景下的解释效果，发现注意力权重相关性较高，并且比基于梯度的显著性提取方法更有效。