We introduce SONAR, a new multilingual and multimodal fixed-size sentence
embedding space. Our single text encoder, covering 200 languages, substantially
outperforms existing sentence embeddings such as LASER3 and LabSE on the xsim
and xsim++ multilingual similarity search tasks. Speech segments can be
embedded in the same SONAR embedding space using language-specific speech
encoders trained in a teacher-student setting on speech transcription data. Our
encoders outperform existing speech encoders on similarity search tasks. We
also provide a text decoder for 200 languages, which allows us to perform
text-to-text and speech-to-text machine translation, including for zero-shot
language and modality combinations. Our text-to-text results are competitive
compared to the state-of-the-art NLLB~1B model, despite the fixed-size
bottleneck representation. Our zero-shot speech-to-text translation results
compare favorably with strong supervised baselines such as Whisper.

我们引入了 SONAR，一个新的多语言和多模态的固定大小句子嵌入空间，在 xsim 和 xsim++ 多语言相似性搜索任务中，我们的单一文本编码器涵盖 200 种语言，表现明显优于现有的句子嵌入如 LASER3 和 LabSE。我们还提供了一个文本解码器，涵盖 200 种语言，可以执行文本到文本和语音到文本的机器翻译，包括零射语言和模态组合。尽管固定大小的瓶颈表示，我们的文本到文本结果与最先进的 NLLB~1B 模型相媲美，我们的零射语音到文本翻译结果与强大的有监督基线如 Whisper 相比也表现得出色。