Previous studies have shown that initializing neural machine translation
(NMT) models with the pre-trained language models (LM) can speed up the model
training and boost the model performance. In this work, we identify a critical
side-effect of pre-training for NMT, which is due to the discrepancy between
the training objectives of LM-based pre-training and NMT. Since the LM
objective learns to reconstruct a few source tokens and copy most of them, the
pre-training initialization would affect the copying behaviors of NMT models.
We provide a quantitative analysis of copying behaviors by introducing a metric
called copying ratio, which empirically shows that pre-training based NMT
models have a larger copying ratio than the standard one. In response to this
problem, we propose a simple and effective method named copying penalty to
control the copying behaviors in decoding. Extensive experiments on both
in-domain and out-of-domain benchmarks show that the copying penalty method
consistently improves translation performance by controlling copying behaviors
for pre-training based NMT models. Source code is freely available at
this https URL

本文研究了使用预训练语言模型来初始化神经机器翻译模型的好处，在发现预训练对机器翻译的副作用时，提出了一种名为 “copying penalty” 的方法来控制解码过程中的复制行为，实验证明，该方法可以提高预训练的机器翻译模型的翻译性能。