In healthcare, making the best possible predictions with complex models
(e.g., neural networks, ensembles/stacks of different models) can impact
patient welfare. In order to make these complex models explainable, we present
DeepSHAP for mixed model types, a framework for layer wise propagation of
Shapley values that builds upon DeepLIFT (an existing approach for explaining
neural networks). We show that in addition to being able to explain neural
networks, this new framework naturally enables attributions for stacks of mixed
models (e.g., neural network feature extractor into a tree model) as well as
attributions of the loss. Finally, we theoretically justify a method for
obtaining attributions with respect to a background distribution (under a
Shapley value framework).

介绍了一种名为 DeepSHAP 的混合模型解释框架，它基于 Shapley 值，可以用来解释由不同模型堆叠而成的复杂模型，包括神经网络和树形模型，并能够为损失函数提供解释。

通过传播局部组成部分的 Shapley 值来解释模型

Explaining Models by Propagating Shapley Values of Local Components

A recent trend in IR has been the usage of neural networks to learn retrieval
models for text based adhoc search. While various approaches and architectures
have yielded significantly better performance than traditional retrieval models
such as BM25, it is still difficult to understand exactly why a document is
relevant to a query. In the ML community several approaches for explaining
decisions made by deep neural networks have been proposed -- including DeepSHAP
which modifies the DeepLift algorithm to estimate the relative importance
(shapley values) of input features for a given decision by comparing the
activations in the network for a given image against the activations caused by
a reference input. In image classification, the reference input tends to be a
plain black image. While DeepSHAP has been well studied for image
classification tasks, it remains to be seen how we can adapt it to explain the
output of Neural Retrieval Models (NRMs). In particular, what is a good "black"
image in the context of IR? In this paper we explored various reference input
document construction techniques. Additionally, we compared the explanations
generated by DeepSHAP to LIME (a model agnostic approach) and found that the
explanations differ considerably. Our study raises concerns regarding the
robustness and accuracy of explanations produced for NRMs. With this paper we
aim to shed light on interesting problems surrounding interpretability in NRMs
and highlight areas of future work.

该研究探索了在信息检索上使用基于神经网络的检索模型时解释方法的有效性，并比较了 DeepSHAP 和 LIME 两种不同解释方法。该研究提出了解释模型的可解释性问题，并针对未来研究的方向提出了建议。