Large-scale models for learning fixed-dimensional cross-lingual sentence
representations like LASER (Artetxe and Schwenk, 2019b) lead to significant
improvement in performance on downstream tasks. However, further increases and
modifications based on such large-scale models are usually impractical due to
memory limitations. In this work, we introduce a lightweight dual-transformer
architecture with just 2 layers for generating memory-efficient cross-lingual
sentence representations. We explore different training tasks and observe that
current cross-lingual training tasks leave a lot to be desired for this shallow
architecture. To ameliorate this, we propose a novel cross-lingual language
model, which combines the existing single-word masked language model with the
newly proposed cross-lingual token-level reconstruction task. We further
augment the training task by the introduction of two computationally-lite
sentence-level contrastive learning tasks to enhance the alignment of
cross-lingual sentence representation space, which compensates for the learning
bottleneck of the lightweight transformer for generative tasks. Our comparisons
with competing models on cross-lingual sentence retrieval and multilingual
document classification confirm the effectiveness of the newly proposed
training tasks for a shallow model.

本文介绍了一种轻量级双变压器体系结构，用于生成记忆高效的跨语言句子表示。同时，还提出了一种新的跨语言语言模型，并引入了两个计算良好的句子级对比学习任务，以提高跨语言句子表示空间的对齐度，从而补偿生成任务的学习瓶颈。实验结果表明，在跨语言句子检索和多语言文档分类方面，与竞争模型相比，我们提出的新的训练任务有效性更高。