Existing zero-shot cross-lingual transfer methods rely on parallel corpora or
bilingual dictionaries, which are expensive and impractical for low-resource
languages. To disengage from these dependencies, researchers have explored
training multilingual models on English-only resources and transferring them to
low-resource languages. However, its effect is limited by the gap between
embedding clusters of different languages. To address this issue, we propose
Embedding-Push, Attention-Pull, and Robust targets to transfer English
embeddings to virtual multilingual embeddings without semantic loss, thereby
improving cross-lingual transferability. Experimental results on mBERT and
XLM-R demonstrate that our method significantly outperforms previous works on
the zero-shot cross-lingual text classification task and can obtain a better
multilingual alignment.

本文提出一种无监督的跨语言嵌入转换方法，其中使用 Embedding-Push、Attention-Pull 和 Robust targets 来处理语言嵌入之间的聚类差异，以提高跨语言转换的可靠性。 实验结果表明，该方法在零 - shot 跨语言文本分类任务上取得显着优于以往的工作，可以获得更好的多语言对齐。