The Transformer model has achieved state-of-the-art performance in many sequence modeling tasks. However, how to leverage model capacity with large or variable depths is still an open challenge. We present a probabilistic framework to automatically learn which layer(s) to use by learning the posterior distributions of layer selection. As an extension of this framework, we propose a novel method to train one shared Transformer network for multilingual machine translation with different layer selection posteriors for each language pair. The proposed method alleviates the vanishing gradient issue and enables stable training of deep Transformers (e.g. 100 layers). We evaluate on WMT English-German machine translation and masked language modeling tasks, where our method outperforms existing approaches for training deeper Transformers. Experiments on multilingual machine translation demonstrate that this approach can effectively leverage increased model capacity and bring universal improvement for both many-to-one and one-to-many translation with diverse language pairs.

本研究提出一种概率框架来自动学习选择哪些层来使用以优化Transformer模型的性能，用于训练一种适用于多语言机器翻译的共享Transformer网络，缓解梯度消失问题，从而使得深度Transformer网络（例如100层）稳定训练，并在机器翻译和语言建模任务中实现了更好的性能。

深度变换器与潜在深度