Recently, diffusion models have excelled in image generation tasks and have
also been applied to neural language processing (NLP) for controllable text
generation. However, the application of diffusion models in a cross-lingual
setting is less unexplored. Additionally, while pretraining with diffusion
models has been studied within a single language, the potential of
cross-lingual pretraining remains understudied. To address these gaps, we
propose XDLM, a novel Cross-lingual diffusion model for machine translation,
consisting of pretraining and fine-tuning stages. In the pretraining stage, we
propose TLDM, a new training objective for mastering the mapping between
different languages; in the fine-tuning stage, we build up the translation
system based on the pretrained model. We evaluate the result on several machine
translation benchmarks and outperformed both diffusion and Transformer
baselines.

提出了 XDLM，一种新颖的用于机器翻译的跨语言扩散模型，包括预训练和微调阶段。在预训练阶段，提出了 TLDM，用于掌握不同语言之间的映射关系；在微调阶段，基于预训练模型构建了翻译系统，并在多个机器翻译基准测试中优于扩散模型和 Transformer 基线。

XDLM：跨语言扩散语言模型用于机器翻译

XDLM: Cross-lingual Diffusion Language Model for Machine Translation

Successful methods for unsupervised neural machine translation (UNMT) employ
crosslingual pretraining via self-supervision, often in the form of a masked
language modeling or a sequence generation task, which requires the model to
align the lexical- and high-level representations of the two languages. While
cross-lingual pretraining works for similar languages with abundant corpora, it
performs poorly in low-resource and distant languages. Previous research has
shown that this is because the representations are not sufficiently aligned. In
this paper, we enhance the bilingual masked language model pretraining with
lexical-level information by using type-level cross-lingual subword embeddings.
Empirical results demonstrate improved performance both on UNMT (up to 4.5
BLEU) and bilingual lexicon induction using our method compared to a UNMT
baseline.

本文介绍了一种基于子词嵌入的双语掩码语言模型预训练方法，应用于无监督神经机器翻译和双语词汇归纳任务中均取得了较好的性能表现。

针对无监督神经机器翻译，提升预训练语言模型的词汇能力

Improving the Lexical Ability of Pretrained Language Models for  Unsupervised Neural Machine Translation

Recent studies have demonstrated the efficiency of generative pretraining for
English natural language understanding. In this work, we extend this approach
to multiple languages and show the effectiveness of cross-lingual pretraining.
We propose two methods to learn cross-lingual language models (XLMs): one
unsupervised that only relies on monolingual data, and one supervised that
leverages parallel data with a new cross-lingual language model objective. We
obtain state-of-the-art results on cross-lingual classification, unsupervised
and supervised machine translation. On XNLI, our approach pushes the state of
the art by an absolute gain of 4.9% accuracy. On unsupervised machine
translation, we obtain 34.3 BLEU on WMT'16 German-English, improving the
previous state of the art by more than 9 BLEU. On supervised machine
translation, we obtain a new state of the art of 38.5 BLEU on WMT'16
Romanian-English, outperforming the previous best approach by more than 4 BLEU.
Our code and pretrained models will be made publicly available.

本文提出了两种跨语言学习模型的方法 (XLMs): 一种是仅依赖于单语数据的无监督方式， 另一种是利用新的跨语言模型目标并使用平行数据的有监督方式。通过这些方法在跨语言分类、无监督和有监督机器翻译中取得了最先进的结果。