The general trend in NLP is towards increasing model capacity and performance
via deeper neural networks. However, simply stacking more layers of the popular
Transformer architecture for machine translation results in poor convergence
and high computational overhead. Our empirical analysis suggests that
convergence is poor due to gradient vanishing caused by the interaction between
residual connections and layer normalization. We propose depth-scaled
initialization (DS-Init), which decreases parameter variance at the
initialization stage, and reduces output variance of residual connections so as
to ease gradient back-propagation through normalization layers. To address
computational cost, we propose a merged attention sublayer (MAtt) which
combines a simplified averagebased self-attention sublayer and the
encoderdecoder attention sublayer on the decoder side. Results on WMT and IWSLT
translation tasks with five translation directions show that deep Transformers
with DS-Init and MAtt can substantially outperform their base counterpart in
terms of BLEU (+1.1 BLEU on average for 12-layer models), while matching the
decoding speed of the baseline model thanks to the efficiency improvements of
MAtt.

本文研究了 NLP 中 Transformers 的深度网络结构存在的一些问题，提出了使用 DS-Init 和 MAtt 方法来解决梯度消失和计算效率问题，实验证明这些方法可以有效提升机器翻译的 BLEU 值。