Multilingual pre-training significantly improves many multilingual NLP tasks,
including machine translation. Most existing methods are based on some variants
of masked language modeling and text-denoising objectives on monolingual data.
Multilingual pre-training on monolingual data ignores the availability of
parallel data in many language pairs. Also, some other works integrate the
available human-generated parallel translation data in their pre-training. This
kind of parallel data is definitely helpful, but it is limited even in
high-resource language pairs. This paper introduces a novel semi-supervised
method, SPDG, that generates high-quality pseudo-parallel data for multilingual
pre-training. First, a denoising model is pre-trained on monolingual data to
reorder, add, remove, and substitute words, enhancing the pre-training
documents' quality. Then, we generate different pseudo-translations for each
pre-training document using dictionaries for word-by-word translation and
applying the pre-trained denoising model. The resulting pseudo-parallel data is
then used to pre-train our multilingual sequence-to-sequence model, PEACH. Our
experiments show that PEACH outperforms existing approaches used in training
mT5 and mBART on various translation tasks, including supervised, zero- and
few-shot scenarios. Moreover, PEACH's ability to transfer knowledge between
similar languages makes it particularly useful for low-resource languages. Our
results demonstrate that with high-quality dictionaries for generating accurate
pseudo-parallel, PEACH can be valuable for low-resource languages.

本研究介绍一种新的半监督方法，能为多语言预训练生成高质量的伪平行数据，结合高质量的字典，能够在机器翻译等多种任务中胜过现有方法，特别适用于低资源语言。

PEACH：预训练序列到序列的多语言翻译模型，使用半监督伪并行文档生成

PEACH: Pre-Training Sequence-to-Sequence Multilingual Models for  Translation with Semi-Supervised Pseudo-Parallel Document Generation

The vast majority of evaluation metrics for machine translation are
supervised, i.e., (i) are trained on human scores, (ii) assume the existence of
reference translations, or (iii) leverage parallel data. This hinders their
applicability to cases where such supervision signals are not available. In
this work, we develop fully unsupervised evaluation metrics. To do so, we
leverage similarities and synergies between evaluation metric induction,
parallel corpus mining, and MT systems. In particular, we use an unsupervised
evaluation metric to mine pseudo-parallel data, which we use to remap deficient
underlying vector spaces (in an iterative manner) and to induce an unsupervised
MT system, which then provides pseudo-references as an additional component in
the metric. Finally, we also induce unsupervised multilingual sentence
embeddings from pseudo-parallel data. We show that our fully unsupervised
metrics are effective, i.e., they beat supervised competitors on 4 out of our 5
evaluation datasets. We make our code publicly available.

本研究开发了全无监督评估指标，利用评估指标的彼此相似性和协同作用、平行语料库挖掘和机器翻译系统等方法，以及迭代地挖掘伪平行数据、对不足的基础矢量空间重新映射并诱导无监督机器翻译系统，最终提供伪参考作为度量的附加组件，同时还利用伪平行数据诱导无监督多语言句子嵌入。结果表明，我们的全无监督度量是有效的，在 5 个数据集中的 4 个上打败了监督竞争对手。

USCORE: 机器翻译完全无监督评价度量的有效方法

USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translation

Back-translation is an effective strategy to improve the performance of
Neural Machine Translation~(NMT) by generating pseudo-parallel data. However,
several recent works have found that better translation quality of the
pseudo-parallel data does not necessarily lead to better final translation
models, while lower-quality but more diverse data often yields stronger
results. In this paper, we propose a novel method to generate pseudo-parallel
data from a pre-trained back-translation model. Our method is a meta-learning
algorithm which adapts a pre-trained back-translation model so that the
pseudo-parallel data it generates would train a forward-translation model to do
well on a validation set. In our evaluations in both the standard datasets WMT
En-De'14 and WMT En-Fr'14, as well as a multilingual translation setting, our
method leads to significant improvements over strong baselines. Our code will
be made available.

本文提出了一种基于元学习的算法，通过调整预先训练的 back-translation 模型来生成伪平行数据，用于训练前向翻译模型，并在 WMT 数据集和多语言翻译中实现了显著的改进。