Text embeddings are useful features for several NLP applications, such as
sentence similarity, text clustering, and semantic search. In this paper, we
present a Low-rank Adaptation with a Contrastive objective on top of 8-bit
Siamese-BLOOM, a multilingual large language model optimized to produce
semantically meaningful word embeddings. The innovation is threefold. First, we
cast BLOOM weights to 8-bit values. Second, we fine-tune BLOOM with a scalable
adapter (LoRA) and 8-bit Adam optimizer for sentence similarity classification.
Third, we apply a Siamese architecture on BLOOM model with a contrastive
objective to ease the multi-lingual labeled data scarcity. The experiment
results show the quality of learned embeddings from LACoS-BLOOM is proportional
to the number of model parameters and the amount of unlabeled training data.
With the parameter efficient fine-tuning design, we are able to run BLOOM 7.1
billion parameters end-to-end on a single GPU machine with 32GB memory.
Compared to previous solution Sentence-BERT, we achieve significant improvement
on both English and multi-lingual STS tasks.

本文介绍了一种低秩适配器、基于对比目标的多语言大型语言模型 Siamese-BLOOM 和可扩展适配器 LoRA 以及 8 位 Adam 优化器，用于句子相似性分类。通过在 BLOOM 模型上应用对比目标的 Siamese 架构，解决了多语言标记数据稀缺的问题，并证明了从 LACoS-BLOOM 中学习到的嵌入质量与模型参数数量和未标记的训练数据量成比例，相对于以前的解决方案 Sentence-BERT，在英语和多语言 STS 任务上均取得了显着的改善。