The multilingual Sentence-BERT (SBERT) models map different languages to
common representation space and are useful for cross-language similarity and
mining tasks. We propose a simple yet effective approach to convert vanilla
multilingual BERT models into multilingual sentence BERT models using synthetic
corpus. We simply aggregate translated NLI or STS datasets of the low-resource
target languages together and perform SBERT-like fine-tuning of the vanilla
multilingual BERT model. We show that multilingual BERT models are inherent
cross-lingual learners and this simple baseline fine-tuning approach without
explicit cross-lingual training yields exceptional cross-lingual properties. We
show the efficacy of our approach on 10 major Indic languages and also show the
applicability of our approach to non-Indic languages German and French. Using
this approach, we further present L3Cube-IndicSBERT, the first multilingual
sentence representation model specifically for Indian languages Hindi, Marathi,
Kannada, Telugu, Malayalam, Tamil, Gujarati, Odia, Bengali, and Punjabi. The
IndicSBERT exhibits strong cross-lingual capabilities and performs
significantly better than alternatives like LaBSE, LASER, and
paraphrase-multilingual-mpnet-base-v2 on Indic cross-lingual and monolingual
sentence similarity tasks. We also release monolingual SBERT models for each of
the languages and show that IndicSBERT performs competitively with its
monolingual counterparts. These models have been evaluated using embedding
similarity scores and classification accuracy.

该论文提出了一种用于转换 vanilla multilingual BERT 模型成多语言句子 BERT 模型的简单有效方法，该方法使用合成语料库聚合并细调 vanilla multilingual BERT 模型的翻译 NLI 或 STS 数据集，展示了多语言 BERT 模型的跨语言学习能力，该方法在 10 种主要的印度语言以及德语和法语这些非印欧语系的语言上均取得了很好的效果，进而实现了首个针对印度语言 Hindi、Marathi、Kannada、Telugu、Malayalam、Tamil、Gujarati、Odia、Bengali 和 Punjabi 的多语言句子表示模型。

L3Cube-IndicSBERT：使用多语言 BERT 学习跨语言句子表示的简单方法

L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence  representations using multilingual BERT

The lack of labeled data is a major obstacle to learning high-quality
sentence embeddings. Recently, self-supervised contrastive learning (SCL) is
regarded as a promising way to address this problem. However, the existing
works mainly rely on hand-crafted data annotation heuristics to generate
positive training samples, which not only call for domain expertise and
laborious tuning, but are also prone to the following unfavorable cases: 1)
trivial positives, 2) coarse-grained positives, and 3) false positives. As a
result, the self-supervision's quality can be severely limited in reality. In
this work, we propose a novel framework InfoCSE to address the above problems.
Instead of relying on annotation heuristics defined by humans, it leverages the
sentence representation model itself and realizes the following iterative
self-supervision process: on one hand, the improvement of sentence
representation may contribute to the quality of data annotation; on the other
hand, more effective data annotation helps to generate high-quality positive
samples, which will further improve the current sentence representation model.
In other words, the representation learning and data annotation become mutually
reinforced, where a strong self-supervision effect can be derived. Extensive
experiments are performed based on three benchmark datasets, where notable
improvements can be achieved against the existing SCL-based methods.

本研究基于自监督对比学习方法，提出了一种名为 InfoCSE 的新型框架，用于解决手工数据注释不可靠的问题，通过迭代的自我监督和数据标注反馈，实现了句子表示模型和数据注释方法的相互协作，有效提高了自监督学习的效果。实验证明，该方法在三个基准数据集上取得了显著的改善。