We show that margin-based bitext mining in a multilingual sentence space can
be applied to monolingual corpora of billions of sentences. We are using ten
snapshots of a curated common crawl corpus (Wenzek et al., 2019) totalling 32.7
billion unique sentences. Using one unified approach for 38 languages, we were
able to mine 4.5 billions parallel sentences, out of which 661 million are
aligned with English. 20 language pairs have more then 30 million parallel
sentences, 112 more then 10 million, and most more than one million, including
direct alignments between many European or Asian languages.
To evaluate the quality of the mined bitexts, we train NMT systems for most
of the language pairs and evaluate them on TED, WMT and WAT test sets. Using
our mined bitexts only and no human translated parallel data, we achieve a new
state-of-the-art for a single system on the WMT'19 test set for translation
between English and German, Russian and Chinese, as well as German/French. In
particular, our English/German system outperforms the best single one by close
to 4 BLEU points and is almost on pair with best WMT'19 evaluation system which
uses system combination and back-translation. We also achieve excellent results
for distant languages pairs like Russian/Japanese, outperforming the best
submission at the 2019 workshop on Asian Translation (WAT).

通过使用边际化双语挖掘技术在多语言句子空间内，我们能够针对百亿级别的单语语料库，挖掘出 45 亿双语句子。使用这些挖掘的数据，我们在 TED、WMT 和 WAT 的测试集上，超越了 WMT'19 的最佳结果，并且对于远程语言对（如俄语 / 日语），我们的表现非常出色。