The evaluation of English text embeddings has transitioned from evaluating a
handful of datasets to broad coverage across many tasks through benchmarks such
as MTEB. However, this is not the case for multilingual text embeddings due to
a lack of available benchmarks. To address this problem, we introduce the
Scandinavian Embedding Benchmark (SEB). SEB is a comprehensive framework that
enables text embedding evaluation for Scandinavian languages across 24 tasks,
10 subtasks, and 4 task categories. Building on SEB, we evaluate more than 26
models, uncovering significant performance disparities between public and
commercial solutions not previously captured by MTEB. We open-source SEB and
integrate it with MTEB, thus bridging the text embedding evaluation gap for
Scandinavian languages.

通过引入 Scandinavian Embedding Benchmark (SEB)，该研究评估了 26 多个模型，揭示了公共和商业解决方案之间的显著性能差异，从而填补了评估斯堪的纳维亚语言文本嵌入的空白。

北欧嵌入基准：多语言和单语文本嵌入的全面评估

The Scandinavian Embedding Benchmarks: Comprehensive Assessment of  Multilingual and Monolingual Text Embedding

Contrastive learning has been successfully used for retrieval of semantically
aligned sentences, but it often requires large batch sizes or careful
engineering to work well. In this paper, we instead propose a generative model
for learning multilingual text embeddings which can be used to retrieve or
score sentence pairs. Our model operates on parallel data in $N$ languages and,
through an approximation we introduce, efficiently encourages source separation
in this multilingual setting, separating semantic information that is shared
between translations from stylistic or language-specific variation. We show
careful large-scale comparisons between contrastive and generation-based
approaches for learning multilingual text embeddings, a comparison that has not
been done to the best of our knowledge despite the popularity of these
approaches. We evaluate this method on a suite of tasks including semantic
similarity, bitext mining, and cross-lingual question retrieval -- the last of
which we introduce in this paper. Overall, our Variational Multilingual
Source-Separation Transformer (VMSST) model outperforms both a strong
contrastive and generative baseline on these tasks.

本文提出了一种新的多语种文本嵌入生成模型 —— VMSST，通过在 $N$ 种语言的并行数据上运行，通过一个引入的近似算法，在多语种语境中鼓励源分离，我们在比较学习多语种文本嵌入的对比和基于生成的方法时，对其进行了仔细的大规模比较，并对其进行了语义相似性、位文本挖掘、跨语言问题检索等任务的评估，从而证明了其优越性。