Language Generation Models produce words based on the previous context.
Although existing methods offer input attributions as explanations for a
model's prediction, it is still unclear how prior words affect the model's
decision throughout the layers. In this work, we leverage recent advances in
explainability of the Transformer and present a procedure to analyze models for
language generation. Using contrastive examples, we compare the alignment of
our explanations with evidence of the linguistic phenomena, and show that our
method consistently aligns better than gradient-based and perturbation-based
baselines. Then, we investigate the role of MLPs inside the Transformer and
show that they learn features that help the model predict words that are
grammatically acceptable. Lastly, we apply our method to Neural Machine
Translation models, and demonstrate that they generate human-like source-target
alignments for building predictions.

本文使用 Transformer 的最新可解释性进展提出了分析语言生成模型的过程，并使用对比实例比较了我们的方法与渐变和扰动基线的解释的一致性，结果表明我们的方法具有更好的表现。最后，我们将该方法应用于神经机器翻译模型，并展示了生成人类相似的源 - 目标对齐结果的能力。