Algorithmic approaches to interpreting machine learning models have
proliferated in recent years. We carry out human subject tests that are the
first of their kind to isolate the effect of algorithmic explanations on a key
aspect of model interpretability, simulatability, while avoiding important
confounding experimental factors. A model is simulatable when a person can
predict its behavior on new inputs. Through two kinds of simulation tests
involving text and tabular data, we evaluate five explanations methods: (1)
LIME, (2) Anchor, (3) Decision Boundary, (4) a Prototype model, and (5) a
Composite approach that combines explanations from each method. Clear evidence
of method effectiveness is found in very few cases: LIME improves
simulatability in tabular classification, and our Prototype method is effective
in counterfactual simulation tests. We also collect subjective ratings of
explanations, but we do not find that ratings are predictive of how helpful
explanations are. Our results provide the first reliable and comprehensive
estimates of how explanations influence simulatability across a variety of
explanation methods and data domains. We show that (1) we need to be careful
about the metrics we use to evaluate explanation methods, and (2) there is
significant room for improvement in current methods. All our supporting code,
data, and models are publicly available at:
this https URL

本文研究机器学习模型的解释方法对模型可解释性方面的影响，通过两类涉及文本和表格数据的模拟测试，评估了五种解释方法的有效性，表明模型可模拟性在分类表格方面使用 LIME 可以得到改善，而我们的 Prototype 方法在反事实的模拟测试中也是有效的。本文的结果显示，我们需要对评估解释方法的指标进行谨慎处理，并且当前方法仍有很大的改进空间。