Explainable artificial intelligence (XAI) holds immense significance in
enhancing the deep neural network's transparency and credibility, particularly
in some risky and high-cost scenarios, like synthetic aperture radar (SAR).
Shapley is a game-based explanation technique with robust mathematical
foundations. However, Shapley assumes that model's features are independent,
rendering Shapley explanation invalid for high dimensional models. This study
introduces a manifold-based Shapley method by projecting high-dimensional
features into low-dimensional manifold features and subsequently obtaining
Fusion-Shap, which aims at (1) addressing the issue of erroneous explanations
encountered by traditional Shap; (2) resolving the challenge of
interpretability that traditional Shap faces in complex scenarios.

在合成孔径雷达（SAR）等风险高、成本高的场景中，可解释的人工智能（XAI）对于提高深度神经网络的透明度和可信度至关重要。Shapley 是一种具有强大数学基础的基于博弈的解释技术。然而，Shapley 假设模型的特征是独立的，这使得高维模型的 Shapley 解释无效。本研究通过将高维特征投影到低维流形特征，并随后获得 Fusion-Shap，引入了一种基于流形的 Shapley 方法，旨在（1）解决传统 Shap 遇到的错误解释问题；（2）解决传统 Shap 在复杂场景中面临的可解释性挑战。

基于流形的 SAR 识别网络解释的 Shapley 值

Manifold-based Shapley for SAR Recognization Network Explanation

Some recent works observed the instability of post-hoc explanations when
input side perturbations are applied to the model. This raises the interest and
concern in the stability of post-hoc explanations. However, the remaining
question is: is the instability caused by the neural network model or the
post-hoc explanation method? This work explores the potential source that leads
to unstable post-hoc explanations. To separate the influence from the model, we
propose a simple output probability perturbation method. Compared to prior
input side perturbation methods, the output probability perturbation method can
circumvent the neural model's potential effect on the explanations and allow
the analysis on the explanation method. We evaluate the proposed method with
three widely-used post-hoc explanation methods (LIME (Ribeiro et al., 2016),
Kernel Shapley (Lundberg and Lee, 2017a), and Sample Shapley (Strumbelj and
Kononenko, 2010)). The results demonstrate that the post-hoc methods are
stable, barely producing discrepant explanations under output probability
perturbations. The observation suggests that neural network models may be the
primary source of fragile explanations.

该研究探讨了导致后续解释不稳定的潜在因素，提出了一种输出概率扰动的方法以分离模型影响，验证了 LIME、Kernel Shapley 和 Sample Shapley 三种解释方法的稳定性，并指出神经网络模型可能是解释不稳定的主要原因。