Being one of the IR-NAT (Iterative-refinemennt-based NAT) frameworks, the
Conditional Masked Language Model (CMLM) adopts the mask-predict paradigm to
re-predict the masked low-confidence tokens. However, CMLM suffers from the
data distribution discrepancy between training and inference, where the
observed tokens are generated differently in the two cases. In this paper, we
address this problem with the training approaches of error exposure and
consistency regularization (EECR). We construct the mixed sequences based on
model prediction during training, and propose to optimize over the masked
tokens under imperfect observation conditions. We also design a consistency
learning method to constrain the data distribution for the masked tokens under
different observing situations to narrow down the gap between training and
inference. The experiments on five translation benchmarks obtains an average
improvement of 0.68 and 0.40 BLEU scores compared to the base models,
respectively, and our CMLMC-EECR achieves the best performance with a
comparable translation quality with the Transformer. The experiments results
demonstrate the effectiveness of our method.

通过错误曝光和一致性正则化的训练方法，研究中提出了解决条件掩码语言模型 (CMLM) 在训练和推断之间数据分布差异的问题，并在五个翻译基准上实验，取得了相对基线模型平均 0.68 和 0.40 BLEU 得分的改进，同时我们的 CMLMC-EECR 在与 Transformer 相比的翻译质量上达到了最佳表现，实验结果验证了我们方法的有效性。