Recent efforts to improve the interpretability of deep neural networks use
saliency to characterize the importance of input features to predictions made
by models. Work on interpretability using saliency-based methods on Recurrent
Neural Networks (RNNs) has mostly targeted language tasks, and their
applicability to time series data is less understood. In this work we analyze
saliency-based methods for RNNs, both classical and gated cell architectures.
We show that RNN saliency vanishes over time, biasing detection of salient
features only to later time steps and are, therefore, incapable of reliably
detecting important features at arbitrary time intervals. To address this
vanishing saliency problem, we propose a novel RNN cell structure (input-cell
attention), which can extend any RNN cell architecture. At each time step,
instead of only looking at the current input vector, input-cell attention uses
a fixed-size matrix embedding, each row of the matrix attending to different
inputs from current or previous time steps. Using synthetic data, we show that
the saliency map produced by the input-cell attention RNN is able to faithfully
detect important features regardless of their occurrence in time. We also apply
the input-cell attention RNN on a neuroscience task analyzing functional
Magnetic Resonance Imaging (fMRI) data for human subjects performing a variety
of tasks. In this case, we use saliency to characterize brain regions (input
features) for which activity is important to distinguish between tasks. We show
that standard RNN architectures are only capable of detecting important brain
regions in the last few time steps of the fMRI data, while the input-cell
attention model is able to detect important brain region activity across time
without latter time step biases.

通过分析神经网络的显著性与循环神经网络的结构，我们发现传统结构不能解决时间上显著性消失的问题，我们提出了一种新的输入 - 细胞注意力 RNN 结构，它能够在任意时刻检测到重要特征，不仅仅在后来的时间步中，这对于 FMRI 数据分析等任务具有广泛的应用。