Deep encoders have been proven to be effective in improving neural machine
translation (NMT) systems, but training an extremely deep encoder is time
consuming. Moreover, why deep models help NMT is an open question. In this
paper, we investigate the behavior of a well-tuned deep Transformer system. We
find that stacking layers is helpful in improving the representation ability of
NMT models and adjacent layers perform similarly. This inspires us to develop a
shallow-to-deep training method that learns deep models by stacking shallow
models. In this way, we successfully train a Transformer system with a 54-layer
encoder. Experimental results on WMT'16 English-German and WMT'14
English-French translation tasks show that it is $1.4$ $\times$ faster than
training from scratch, and achieves a BLEU score of $30.33$ and $43.29$ on two
tasks. The code is publicly available at
this https URL

本文研究了通过浅层模型堆叠训练深度编码器的浅层到深层的方法对神经机器翻译模型进行改进，并通过实验表明该方法可以显著提高翻译质量，并在 WMT'16 和 WMT'14 的任务中取得了 30.33 和 43.29 的 BLEU 分数。