Unsupervised on-the-fly back-translation, in conjunction with multilingual
pretraining, is the dominant method for unsupervised neural machine
translation. Theoretically, however, the method should not work in general. We
therefore conduct controlled experiments with artificial languages to determine
what properties of languages make back-translation an effective training
method, covering lexical, syntactic, and semantic properties. We find, contrary
to popular belief, that (i) parallel word frequency distributions, (ii)
partially shared vocabulary, and (iii) similar syntactic structure across
languages are not sufficient to explain the success of back-translation. We
show however that even crude semantic signal (similar lexical fields across
languages) does improve alignment of two languages through back-translation. We
conjecture that rich semantic dependencies, parallel across languages, are at
the root of the success of unsupervised methods based on back-translation.
Overall, the success of unsupervised machine translation was far from being
analytically guaranteed. Instead, it is another proof that languages of the
world share deep similarities, and we hope to show how to identify which of
these similarities can serve the development of unsupervised, cross-linguistic
tools.

无监督神经机器翻译中的实时反向翻译与多语言预训练是主要方法，该论文通过人工语言的实验，发现词汇、句法和语义属性并不能充分解释反向翻译成功的原因，但是语义信号能通过反向翻译改进两种语言的对齐，推测多语言间存在平行的丰富语义依赖，这证明了无监督机器翻译的成功并非分析性的保证，并希望能够确定哪些共同性可用于无监督、跨语言工具的开发。

句法和语义接近度对机器翻译及逆向翻译的影响

The Impact of Syntactic and Semantic Proximity on Machine Translation  with Back-Translation

Since language models are used to model a wide variety of languages, it is
natural to ask whether the neural architectures used for the task have
inductive biases towards modeling particular types of languages. Investigation
of these biases has proved complicated due to the many variables that appear in
the experimental setup. Languages vary in many typological dimensions, and it
is difficult to single out one or two to investigate without the others acting
as confounders. We propose a novel method for investigating the inductive
biases of language models using artificial languages. These languages are
constructed to allow us to create parallel corpora across languages that differ
only in the typological feature being investigated, such as word order. We then
use them to train and test language models. This constitutes a fully controlled
causal framework, and demonstrates how grammar engineering can serve as a
useful tool for analyzing neural models. Using this method, we find that
commonly used neural architectures exhibit different inductive biases: LSTMs
display little preference with respect to word ordering, while transformers
display a clear preference for some orderings over others. Further, we find
that neither the inductive bias of the LSTM nor that of the transformer appears
to reflect any tendencies that we see in attested natural languages.

该研究提出了一种新颖的方法，使用人造语言验证语言模型的归纳偏见，并发现常用的神经架构在处理语句单词排序方面存在不同的归纳偏差。