Local model interpretation methods explain individual predictions by
assigning an importance value to each input feature. This value is often
determined by measuring the change in confidence when a feature is removed.
However, the confidence of neural networks is not a robust measure of model
uncertainty. This issue makes reliably judging the importance of the input
features difficult. We address this by changing the test-time behavior of
neural networks using Deep k-Nearest Neighbors. Without harming text
classification accuracy, this algorithm provides a more robust uncertainty
metric which we use to generate feature importance values. The resulting
interpretations better align with human perception than baseline methods.
Finally, we use our interpretation method to analyze model predictions on
dataset annotation artifacts.

通过使用深度 K 近邻算法来改变神经网络的测试时间行为，本文提出了一种更健壮的不确定性度量，以生成特征重要性值，并使用这种解释方法来分析数据集注释人工制品上的模型预测。

用最近邻居解释神经网络

Interpreting Neural Networks With Nearest Neighbors

Deep neural networks (DNNs) enable innovative applications of machine
learning like image recognition, machine translation, or malware detection.
However, deep learning is often criticized for its lack of robustness in
adversarial settings (e.g., vulnerability to adversarial inputs) and general
inability to rationalize its predictions. In this work, we exploit the
structure of deep learning to enable new learning-based inference and decision
strategies that achieve desirable properties such as robustness and
interpretability. We take a first step in this direction and introduce the Deep
k-Nearest Neighbors (DkNN). This hybrid classifier combines the k-nearest
neighbors algorithm with representations of the data learned by each layer of
the DNN: a test input is compared to its neighboring training points according
to the distance that separates them in the representations. We show the labels
of these neighboring points afford confidence estimates for inputs outside the
model's training manifold, including on malicious inputs like adversarial
examples--and therein provides protections against inputs that are outside the
models understanding. This is because the nearest neighbors can be used to
estimate the nonconformity of, i.e., the lack of support for, a prediction in
the training data. The neighbors also constitute human-interpretable
explanations of predictions. We evaluate the DkNN algorithm on several
datasets, and show the confidence estimates accurately identify inputs outside
the model, and that the explanations provided by nearest neighbors are
intuitive and useful in understanding model failures.

本研究针对深度学习在对抗性环境下的鲁棒性和预测不可解释性等问题，通过将 k-NN 算法与深度学习结合，提出了一种名为 DkNN 的混合分类器，它可以为输入数据提供信心估计和人类可解释的预测解释。实验证明，DkNN 算法可以准确识别模型外部的输入，同时提供直观和有用的模型失败解释。