Successful methods for unsupervised neural machine translation (UNMT) employ
crosslingual pretraining via self-supervision, often in the form of a masked
language modeling or a sequence generation task, which requires the model to
align the lexical- and high-level representations of the two languages. While
cross-lingual pretraining works for similar languages with abundant corpora, it
performs poorly in low-resource and distant languages. Previous research has
shown that this is because the representations are not sufficiently aligned. In
this paper, we enhance the bilingual masked language model pretraining with
lexical-level information by using type-level cross-lingual subword embeddings.
Empirical results demonstrate improved performance both on UNMT (up to 4.5
BLEU) and bilingual lexicon induction using our method compared to a UNMT
baseline.

本文介绍了一种基于子词嵌入的双语掩码语言模型预训练方法，应用于无监督神经机器翻译和双语词汇归纳任务中均取得了较好的性能表现。

针对无监督神经机器翻译，提升预训练语言模型的词汇能力

Improving the Lexical Ability of Pretrained Language Models for  Unsupervised Neural Machine Translation

Multilingual training of neural machine translation (NMT) systems has led to
impressive accuracy improvements on low-resource languages. However, there are
still significant challenges in efficiently learning word representations in
the face of paucity of data. In this paper, we propose Soft Decoupled Encoding
(SDE), a multilingual lexicon encoding framework specifically designed to share
lexical-level information intelligently without requiring heuristic
preprocessing such as pre-segmenting the data. SDE represents a word by its
spelling through a character encoding, and its semantic meaning through a
latent embedding space shared by all languages. Experiments on a standard
dataset of four low-resource languages show consistent improvements over strong
multilingual NMT baselines, with gains of up to 2 BLEU on one of the tested
languages, achieving the new state-of-the-art on all four language pairs.

本研究提出了一种名为 SDE 的多语言词典编码框架，通过字符编码和语义嵌入空间的共享来实现智能共享词汇级别信息，从而实现在低资源语言中神经机器翻译系统的多语言训练。在标准数据集上的实验表明，与强大的多语言神经机器翻译基线相比，本方法在四种低资源语言中表现出一贯的提高，并在其中一种测试语言上获得了 2 BLEU 的增益，并在所有四种语言对中实现了新的最优状态。