Dense retrieval needs to learn discriminative text embeddings to represent
the semantic relationship between query and document. It may benefit from the
using of large language models (LLMs), given LLMs' strong capability on
semantic understanding. However, the LLMs are pre-trained by text generation
tasks, whose working pattern is completely different from representing texts as
embeddings. As a result, it is imperative to study how to adapt LLMs properly
so that they can be effectively initialized as the backbone encoder for dense
retrieval.
In this paper, we propose a novel approach, called LLaRA (LLM adapted for
dense RetrievAl), which works as a post-hoc adaptation of LLM for the dense
retrieval application. LLaRA consists of two pretext tasks: EBAE
(Embedding-Based Auto-Encoding) and EBAR (Embedding-Based Auto-Regression),
where the text embeddings from LLM are used to reconstruct the tokens for the
input sentence and predict the tokens for the next sentence, respectively.
LLaRA turns out to be simple, lightweight, and highly effective. It is applied
to adapt LLaMA-2-7B (base) on the Wikipedia corpus, where it substantially
improves the model's fine-tuned performances on a variety of dense retrieval
benchmarks, like MSMARCO and BEIR. Our model and code will be made publicly
available at BGE repository.

提出了一种新颖的方法 LLaRA（LLM 适应于密集检索），它作为 LLM 的事后适应工具，用于密集检索应用。LLaRA 包括两个预处理任务：EBAE (基于嵌入的自编码) 和 EBAR (基于嵌入的自回归)，其中来自 LLM 的文本嵌入用于重建输入句子的标记并预测下一句的标记。LLaRA 简单、轻量且高效，应用于 LLMaMA-2-7B（基础）模型，在维基百科语料库上大大提升了模型对各种密集检索基准（如 MSMARCO 和 BEIR）的微调性能。