Pre-training and fine-tuning have achieved great success in the natural
language process field. The standard paradigm of exploiting them includes two
steps: first, pre-training a model, e.g. BERT, with a large scale unlabeled
monolingual data. Then, fine-tuning the pre-trained model with labeled data
from downstream tasks. However, in neural machine translation (NMT), we address
the problem that the training objective of the bilingual task is far different
from the monolingual pre-trained model. This gap leads that only using
fine-tuning in NMT can not fully utilize prior language knowledge. In this
paper, we propose an APT framework for acquiring knowledge from the pre-trained
model to NMT. The proposed approach includes two modules: 1). a dynamic fusion
mechanism to fuse task-specific features adapted from general knowledge into
NMT network, 2). a knowledge distillation paradigm to learn language knowledge
continuously during the NMT training process. The proposed approach could
integrate suitable knowledge from pre-trained models to improve the NMT.
Experimental results on WMT English to German, German to English and Chinese to
English machine translation tasks show that our model outperforms strong
baselines and the fine-tuning counterparts.

本文提出了一种 APT 框架，在神经机器翻译中利用预训练模型的语言知识，该框架包括两个模块：1）动态融合机制，将从通用知识中适应的任务特定特征融合到 NMT 网络中；2）知识蒸馏范式，在 NMT 训练过程中连续学习语言知识。实验结果表明，我们的模型优于强基线和微调对应模型。