Multilingual representations pre-trained with monolingual data exhibit
considerably unequal task performances across languages. Previous studies
address this challenge with resource-intensive contextualized alignment, which
assumes the availability of large parallel data, thereby leaving
under-represented language communities behind. In this work, we attribute the
data hungriness of previous alignment techniques to two limitations: (i) the
inability to sufficiently leverage data and (ii) these techniques are not
trained properly. To address these issues, we introduce supervised and
unsupervised density-based approaches named Real-NVP and GAN-Real-NVP, driven
by Normalizing Flow, to perform alignment, both dissecting the alignment of
multilingual subspaces into density matching and density modeling. We
complement these approaches with our validation criteria in order to guide the
training process. Our experiments encompass 16 alignments, including our
approaches, evaluated across 6 language pairs, synthetic data and 5 NLP tasks.
We demonstrate the effectiveness of our approaches in the scenarios of limited
and no parallel data. First, our supervised approach trained on 20k parallel
data (sentences) mostly surpasses Joint-Align and InfoXLM trained on over 100k
parallel sentences. Second, parallel data can be removed without sacrificing
performance when integrating our unsupervised approach in our bootstrapping
procedure, which is theoretically motivated to enforce equality of multilingual
subspaces. Moreover, we demonstrate the advantages of validation criteria over
validation data for guiding supervised training.

本研究采用密度匹配和密度建模的方法进行多语种词向量的对齐，提出基于 Normalizing Flow 的 Real-NVP 和 GAN-Real-NVP 两种监督学习和无监督学习的密度对齐方法，并通过验证标准来指导训练过程。实验结果表明，我们的方法在有限或没有平行数据的情况下均能显著提高多项 NLP 任务的表现。