One of the motivations for explainable AI is to allow humans to make better
and more informed decisions regarding the use and deployment of AI models. But
careful evaluations are needed to assess whether this expectation has been
fulfilled. Current evaluations mainly focus on algorithmic properties of
explanations, and those that involve human subjects often employ subjective
questions to test human's perception of explanation usefulness, without being
grounded in objective metrics and measurements. In this work, we evaluate
whether explanations can improve human decision-making in practical scenarios
of machine learning model development. We conduct a mixed-methods user study
involving image data to evaluate saliency maps generated by SmoothGrad,
GradCAM, and an oracle explanation on two tasks: model selection and
counterfactual simulation. To our surprise, we did not find evidence of
significant improvement on these tasks when users were provided with any of the
saliency maps, even the synthetic oracle explanation designed to be simple to
understand and highly indicative of the answer. Nonetheless, explanations did
help users more accurately describe the models. These findings suggest caution
regarding the usefulness and potential for misunderstanding in saliency-based
explanations.

通过用户研究，本研究评估了可解释人工智能在实际场景中对人类决策的改进效果，结果发现虽然解释有助于用户更准确地描述模型，但对于模型选择和反事实模拟这两个任务，并没有找到使用任何显著改进的证据，这表明对基于显著性的解释的实用性和可能的误解需要谨慎对待。

评估模型解释在模型开发中的效用

Evaluating the Utility of Model Explanations for Model Development

Despite the rising popularity of saliency-based explanations, the research
community remains at an impasse, facing doubts concerning their purpose,
efficacy, and tendency to contradict each other. Seeking to unite the
community's efforts around common goals, several recent works have proposed
evaluation metrics. In this paper, we critically examine two sets of metrics:
the ERASER metrics (comprehensiveness and sufficiency) and the EVAL-X metrics,
focusing our inquiry on natural language processing. First, we show that we can
inflate a model's comprehensiveness and sufficiency scores dramatically without
altering its predictions or explanations on in-distribution test inputs. Our
strategy exploits the tendency for extracted explanations and their complements
to be "out-of-support" relative to each other and in-distribution inputs. Next,
we demonstrate that the EVAL-X metrics can be inflated arbitrarily by a simple
method that encodes the label, even though EVAL-X is precisely motivated to
address such exploits. Our results raise doubts about the ability of current
metrics to guide explainability research, underscoring the need for a broader
reassessment of what precisely these metrics are intended to capture.

对于基于显著性的解释方法的研究，存在关于其目的、有效性和相互抵触性的疑虑。本文针对自然语言处理进行了关于评估指标的批判性研究，对两套指标进行了评估，并展示了目前的指标能否准确引导可解释性研究的能力存在疑问，强调有必要对这些指标所要捕捉的内容进行更广泛的重新评估。

Goodhart 定律在 NLP 的解释基准中适用

Goodhart's Law Applies to NLP's Explanation Benchmarks

This paper aims to provide a framework to quantitatively evaluate and rank
explanation methods for the time series classification task, which deals with a
prevalent data type in critical domains such as healthcare and finance. The
recent surge of research interest in explanation methods for time series
classification has provided a great variety of explanation techniques.
Nevertheless, when these explanation techniques disagree on a specific problem,
it remains unclear which of them to use. Comparing the explanations to find the
right answer is non-trivial. Two key challenges remain: how to quantitatively
and robustly evaluate the informativeness (i.e., relevance for the
classification task) of a given explanation method, and how to compare
explanation methods side-by-side. We propose AMEE, a Model-Agnostic Explanation
Evaluation framework for quantifying and comparing multiple saliency-based
explanations for time series classification. Perturbation is added to the input
time series guided by the saliency maps (i.e., importance weights for each
point in the time series). The impact of perturbation on classification
accuracy is measured and used for explanation evaluation. The results show that
perturbing discriminative parts of the time series leads to significant changes
in classification accuracy. To be robust to different types of perturbations
and different types of classifiers, we aggregate the accuracy loss across
perturbations and classifiers. This allows us to objectively quantify and rank
different explanation methods. We provide a quantitative and qualitative
analysis for synthetic datasets, a variety of UCR benchmark datasets, as well
as a real-world dataset with known expert ground truth.

本文提出一种名为 AMEE 的模型无关解释评估框架，可以量化和比较用于时序分类的多个显著性解释方法，并通过扰动输入时间序列来测量其对分类准确度的影响以进行解释评估。