We describe an unsupervised method to create pseudo-parallel corpora for
machine translation (MT) from unaligned text. We use multilingual BERT to
create source and target sentence embeddings for nearest-neighbor search and
adapt the model via self-training. We validate our technique by extracting
parallel sentence pairs on the BUCC 2017 bitext mining task and observe up to a
24.5 point increase (absolute) in F1 scores over previous unsupervised methods.
We then improve an XLM-based unsupervised neural MT system pre-trained on
Wikipedia by supplementing it with pseudo-parallel text mined from the same
corpus, boosting unsupervised translation performance by up to 3.5 BLEU on the
WMT'14 French-English and WMT'16 German-English tasks and outperforming the
previous state-of-the-art. Finally, we enrich the IWSLT'15 English-Vietnamese
corpus with pseudo-parallel Wikipedia sentence pairs, yielding a 1.2 BLEU
improvement on the low-resource MT task. We demonstrate that unsupervised
bitext mining is an effective way of augmenting MT datasets and complements
existing techniques like initializing with pre-trained contextual embeddings.

采用无监督学习方法使用多语言 BERT 创建伪平行语料库以提高机器翻译性能，并在不同任务中获得显著提高。

自训练上下文嵌入的无监督双语挖掘与翻译

Unsupervised Bitext Mining and Translation via Self-trained Contextual  Embeddings

In neural machine translation (NMT), the most common practice is to stack a
number of recurrent or feed-forward layers in the encoder and the decoder. As a
result, the addition of each new layer improves the translation quality
significantly. However, this also leads to a significant increase in the number
of parameters. In this paper, we propose to share parameters across all the
layers thereby leading to a recurrently stacked NMT model. We empirically show
that the translation quality of a model that recurrently stacks a single layer
6 times is comparable to the translation quality of a model that stacks 6
separate layers. We also show that using pseudo-parallel corpora by
back-translation leads to further significant improvements in translation
quality.

本文提出跨层参数共享方法，将单层循环堆叠的神经机器翻译模型与完整的六层模型的翻译质量进行比较，结果表明使用伪对称语料库进行反向翻译可以显著提高翻译质量。