Current methods for the interpretability of discriminative deep neural
networks commonly rely on the model's input-gradients, i.e., the gradients of
the output logits w.r.t. the inputs. The common assumption is that these
input-gradients contain information regarding $p_{\theta} ( y \mid x)$, the
model's discriminative capabilities, thus justifying their use for
interpretability. However, in this work we show that these input-gradients can
be arbitrarily manipulated as a consequence of the shift-invariance of softmax
without changing the discriminative function. This leaves an open question: if
input-gradients can be arbitrary, why are they highly structured and
explanatory in standard models?
We investigate this by re-interpreting the logits of standard softmax-based
classifiers as unnormalized log-densities of the data distribution and show
that input-gradients can be viewed as gradients of a class-conditional density
model $p_{\theta}(x \mid y)$ implicit within the discriminative model. This
leads us to hypothesize that the highly structured and explanatory nature of
input-gradients may be due to the alignment of this class-conditional model
$p_{\theta}(x \mid y)$ with that of the ground truth data distribution
$p_{\text{data}} (x \mid y)$. We test this hypothesis by studying the effect of
density alignment on gradient explanations. To achieve this alignment we use
score-matching, and propose novel approximations to this algorithm to enable
training large-scale models.
Our experiments show that improving the alignment of the implicit density
model with the data distribution enhances gradient structure and explanatory
power while reducing this alignment has the opposite effect. Overall, our
finding that input-gradients capture information regarding an implicit
generative model implies that we need to re-think their use for interpreting
discriminative models.

该研究分析了模型 input-gradients 在解释性方面的问题，提出了将标准 softmax-based 分类器的 logits 重新解释为未归一化的数据密度，证明了 input-gradients 可以被视为隐含于判别模型中的类条件密度模型的梯度，并提出了通过 score-matching 来实现对隐含密度模型与数据分布的对齐的算法。研究表明，对齐隐含密度模型和数据分布可以提高梯度的结构性和解释性。