While neural networks have been successfully applied to many natural language processing tasks, they come at the cost of interpretability. In this paper, we propose a general methodology to analyze and interpret decisions from a neural model by observing the effects on the model of erasing various parts of the representation, such as input word-vector dimensions, intermediate hidden units, or input words. We present several approaches to analyzing the effects of such erasure, from computing the relative difference in evaluation metrics, to using reinforcement learning to erase the minimum set of input words in order to flip a neural model's decision. In a comprehensive analysis of multiple NLP tasks, including linguistic feature classification, sentence-level sentiment analysis, and document level sentiment aspect prediction, we show that the proposed methodology not only offers clear explanations about neural model decisions, but also provides a way to conduct error analysis on neural models.

本文提出了一种使用擦除来解释神经模型决策的通用方法，通过分析和比较擦除输入词向量的不同部分，中间隐藏层的不同单元或输入词的不同方法，来评估和解释模型判断的影响。研究表明，该方法不仅可以提供清晰的神经模型决策解释，还可以为神经模型的错误分析提供一种方法。

通过表示擦除理解神经网络