Recent literature has seen growing interest in using black-box strategies
like CheckList for testing the behavior of NLP models. Research on white-box
testing has developed a number of methods for evaluating how thoroughly the
internal behavior of deep models is tested, but they are not applicable to NLP
models. We propose a set of white-box testing methods that are customized for
transformer-based NLP models. These include Mask Neuron Coverage (MNCOVER) that
measures how thoroughly the attention layers in models are exercised during
testing. We show that MNCOVER can refine testing suites generated by CheckList
by substantially reduce them in size, for more than 60\% on average, while
retaining failing tests -- thereby concentrating the fault detection power of
the test suite. Further we show how MNCOVER can be used to guide CheckList
input generation, evaluate alternative NLP testing methods, and drive data
augmentation to improve accuracy.

本论文提出了一种针对基于 transformer 的 NLP 模型的白盒测试方法，其中包括可以衡量模型中的 Attention 层是否得到充分运行的 Mask Neuron Coverage，该方法可用于指导测试数据生成，评估其他 NLP 测试方法，驱动数据增广以提高模型的准确性。