Recent studies have shown that dual encoder models trained with the
sentence-level translation ranking task are effective methods for cross-lingual
sentence embedding. However, our research indicates that token-level alignment
is also crucial in multilingual scenarios, which has not been fully explored
previously. Based on our findings, we propose a dual-alignment pre-training
(DAP) framework for cross-lingual sentence embedding that incorporates both
sentence-level and token-level alignment. To achieve this, we introduce a novel
representation translation learning (RTL) task, where the model learns to use
one-side contextualized token representation to reconstruct its translation
counterpart. This reconstruction objective encourages the model to embed
translation information into the token representation. Compared to other
token-level alignment methods such as translation language modeling, RTL is
more suitable for dual encoder architectures and is computationally efficient.
Extensive experiments on three sentence-level cross-lingual benchmarks
demonstrate that our approach can significantly improve sentence embedding. Our
code is available at this https URL

我们提出了一种双重对齐预训练框架 (DAP)，它结合了句级和令牌级对齐，用于跨语言句子嵌入，其中引入了一种表示翻译学习 (RTL) 任务来将翻译信息嵌入到令牌表示中。在三个跨语言基准测试上进行了广泛实验，表明我们的方法能够显著提高句子嵌入。