Explainable artificial intelligence (XAI) methods are currently evaluated
with approaches mostly originated in interpretable machine learning (IML)
research that focus on understanding models such as comparison against existing
attribution approaches, sensitivity analyses, gold set of features, axioms, or
through demonstration of images. There are problems with these methods such as
that they do not indicate where current XAI approaches fail to guide
investigations towards consistent progress of the field. They do not measure
accuracy in support of accountable decisions, and it is practically impossible
to determine whether one XAI method is better than the other or what the
weaknesses of existing models are, leaving researchers without guidance on
which research questions will advance the field. Other fields usually utilize
ground-truth data and create benchmarks. Data representing ground-truth
explanations is not typically used in XAI or IML. One reason is that
explanations are subjective, in the sense that an explanation that satisfies
one user may not satisfy another. To overcome these problems, we propose to
represent explanations with canonical equations that can be used to evaluate
the accuracy of XAI methods. The contributions of this paper include a
methodology to create synthetic data representing ground-truth explanations,
three data sets, an evaluation of LIME using these data sets, and a preliminary
analysis of the challenges and potential benefits in using these data to
evaluate existing XAI approaches. Evaluation methods based on human-centric
studies are outside the scope of this paper.

本文介绍了使用规范方程代表解释的方法，以解决可解释的人工智能和可解释的机器学习方法面临的挑战，并提出使用合成数据来评估模型准确性的方法。

用于评估 XAI 方法的数据代表真实解释

Data Representing Ground-Truth Explanations to Evaluate XAI Methods

Estimation of the accuracy of a large-scale knowledge graph (KG) often
requires humans to annotate samples from the graph. How to obtain statistically
meaningful estimates for accuracy evaluation while keeping human annotation
costs low is a problem critical to the development cycle of a KG and its
practical applications. Surprisingly, this challenging problem has largely been
ignored in prior research. To address the problem, this paper proposes an
efficient sampling and evaluation framework, which aims to provide quality
accuracy evaluation with strong statistical guarantee while minimizing human
efforts. Motivated by the properties of the annotation cost function observed
in practice, we propose the use of cluster sampling to reduce the overall cost.
We further apply weighted and two-stage sampling as well as stratification for
better sampling designs. We also extend our framework to enable efficient
incremental evaluation on evolving KG, introducing two solutions based on
stratified sampling and a weighted variant of reservoir sampling. Extensive
experiments on real-world datasets demonstrate the effectiveness and efficiency
of our proposed solution. Compared to baseline approaches, our best solutions
can provide up to 60% cost reduction on static KG evaluation and up to 80% cost
reduction on evolving KG evaluation, without loss of evaluation quality.

通过使用簇抽样、加权和两阶段抽样，以及分层的抽样设计，提出了一种有效的抽样和评估框架，该框架旨在在最小化人力成本的同时提供具有强大统计保证的高质量的准确性评估。