There have been significant efforts to interpret the encoder of
Transformer-based encoder-decoder architectures for neural machine translation
(NMT); meanwhile, the decoder remains largely unexamined despite its critical
role. During translation, the decoder must predict output tokens by considering
both the source-language text from the encoder and the target-language prefix
produced in previous steps. In this work, we study how Transformer-based
decoders leverage information from the source and target languages --
developing a universal probe task to assess how information is propagated
through each module of each decoder layer. We perform extensive experiments on
three major translation datasets (WMT En-De, En-Fr, and En-Zh). Our analysis
provides insight on when and where decoders leverage different sources. Based
on these insights, we demonstrate that the residual feed-forward module in each
Transformer decoder layer can be dropped with minimal loss of performance -- a
significant reduction in computation and number of parameters, and consequently
a significant boost to both training and inference speed.

本研究通过开发一种通用的探测任务来分析 Transformer-based 解码器中的每个模块如何传递信息，并针对三个主要翻译数据集进行了广泛实验。通过分析发现，Transformer 解码器中每个模块对应的不同的源信息和目标语前缀的使用方式。同时，通过实验得出，每层解码器中的残差前馈模块可以被取消，而不影响性能，从而极大地降低了计算量和参数数量，提高了培训和推理速度