There are two types of approaches to solving cross-lingual transfer:
multilingual pre-training implicitly aligns the hidden representations of
different languages, while the translate-test explicitly translates different
languages to an intermediate language, such as English. Translate-test has
better interpretability compared to multilingual pre-training. However, the
translate-test has lower performance than multilingual pre-training(Conneau and
Lample, 2019; Conneau et al, 2020) and can't solve word-level tasks because
translation rearranges the word order. Therefore, we propose a new
Machine-created Universal Language (MUL) as a new intermediate language. MUL
consists of a set of discrete symbols as universal vocabulary and NL-MUL
translator for translating from multiple natural languages to MUL. MUL unifies
common concepts from different languages into the same universal word for
better cross-language transfer. And MUL preserves the language-specific words
as well as word order, so the model can be easily applied to word-level tasks.
Our experiments show that translating into MUL achieves better performance
compared to multilingual pre-training, and our analyses show that MUL has good
interpretability.

本文提出了一种新的中间语言，机器创建的通用语言（MUL），将多种自然语言翻译成同一通用词，从而实现更好的跨语言转移和更好的解释性能，实验结果表明，与多语言预训练相比，将其翻译成 MUL 可获得更好的性能，并且 MUL 具有很好的解释性能。