In attempts to "explain" predictions of machine learning models, researchers
have proposed hundreds of techniques for attributing predictions to features
that are deemed important. While these attributions are often claimed to hold
the potential to improve human "understanding" of the models, surprisingly
little work explicitly evaluates progress towards this aspiration. In this
paper, we conduct a crowdsourcing study, where participants interact with
deception detection models that have been trained to distinguish between
genuine and fake hotel reviews. They are challenged both to simulate the model
on fresh reviews, and to edit reviews with the goal of lowering the probability
of the originally predicted class. Successful manipulations would lead to an
adversarial example. During the training (but not the test) phase, input spans
are highlighted to communicate salience. Through our evaluation, we observe
that for a linear bag-of-words model, participants with access to the feature
coefficients during training are able to cause a larger reduction in model
confidence in the testing phase when compared to the no-explanation control.
For the BERT-based classifier, popular local explanations do not improve their
ability to reduce the model confidence over the no-explanation case.
Remarkably, when the explanation for the BERT model is given by the (global)
attributions of a linear model trained to imitate the BERT model, people can
effectively manipulate the model.

该研究通过一个众包实验，探讨解释机器学习模型的预测对人类是否具有帮助，并发现在给定特征系数的情况下，人们能够更有效地对线性词袋模型进行操纵，但解释对 BERT 型分类器并没有显著提高操纵能力，而通过伪造 BERT 模型的线性模型的对全局归属的解释则可以有效地操作 BERT 型模型。