Recently, universal neural machine translation (NMT) with shared encoder-decoder gained good performance on zero-shot translation. Unlike universal NMT, jointly trained language-specific encoders-decoders aim to achieve universal representation across non-shared modules, each of which is for a language or language family. The non-shared architecture has the advantage of mitigating internal language competition, especially when the shared vocabulary and model parameters are restricted in their size. However, the performance of using multiple encoders and decoders on zero-shot translation still lags behind universal NMT. In this work, we study zero-shot translation using language-specific encoders-decoders. We propose to generalize the non-shared architecture and universal NMT by differentiating the Transformer layers between language-specific and interlingua. By selectively sharing parameters and applying cross-attentions, we explore maximizing the representation universality and realizing the best alignment of language-agnostic information. We also introduce a denoising auto-encoding (DAE) objective to jointly train the model with the translation task in a multi-task manner. Experiments on two public multilingual parallel datasets show that our proposed model achieves a competitive or better results than universal NMT and strong pivot baseline. Moreover, we experiment incrementally adding new language to the trained model by only updating the new model parameters. With this little effort, the zero-shot translation between this newly added language and existing languages achieves a comparable result with the model trained jointly from scratch on all languages.

本文介绍了一种利用语言特定的编码器-解码器完成零样本翻译的方法。通过区分语言特定的Transformer层和公共层，应用交叉注意力和参数共享等技术，最大化句子的普适性，以实现最佳的语义对齐，此外结合了自编码目标来实现多任务的联合训练。在两个公共的多语言平行数据集上进行实验，它们的结果具有竞争力，而且相当于从头开始联合训练模型时的结果，还表现出增量学习的能力。

基于语言特定编码器解码器的零样本神经机器翻译改进