Attention mechanisms are dominating the explainability of deep models. They
produce probability distributions over the input, which are widely deemed as
feature-importance indicators. However, in this paper, we find one critical
limitation in attention explanations: weakness in identifying the polarity of
feature impact. This would be somehow misleading -- features with higher
attention weights may not faithfully contribute to model predictions; instead,
they can impose suppression effects. With this finding, we reflect on the
explainability of current attention-based techniques, such as
Attentio$\odot$Gradient and LRP-based attention explanations. We first propose
an actionable diagnostic methodology (henceforth faithfulness violation test)
to measure the consistency between explanation weights and the impact polarity.
Through the extensive experiments, we then show that most tested explanation
methods are unexpectedly hindered by the faithfulness violation issue,
especially the raw attention. Empirical analyses on the factors affecting
violation issues further provide useful observations for adopting explanation
methods in attention models.

本文发现注意力解释的一个关键限制：弱点在于识别特征影响的极性。为此，作者提出了一个行动诊断方法来量化解释权重与影响极性之间的一致性，并通过大量实验表明，大多数测试的解释方法都意外受到忠诚度违反问题的阻碍，特别是原始的关注力。