All text-based language problems can be reduced to either generation or
embedding. Current models only perform well at one or the other. We introduce
generative representational instruction tuning (GRIT) whereby a large language
model is trained to handle both generative and embedding tasks by
distinguishing between them through instructions. Compared to other open
models, our resulting GritLM 7B sets a new state of the art on the Massive Text
Embedding Benchmark (MTEB) and outperforms all models up to its size on a range
of generative tasks. By scaling up further, GritLM 8x7B outperforms all open
generative language models that we tried while still being among the best
embedding models. Notably, we find that GRIT matches training on only
generative or embedding data, thus we can unify both at no performance loss.
Among other benefits, the unification via GRIT speeds up Retrieval-Augmented
Generation (RAG) by > 60% for long documents, by no longer requiring separate
retrieval and generation models. Models, code, etc. are freely available at
this https URL

通过生成性行为表示指导调整（GRIT）技术，我们训练了一个大型语言模型，通过区分生成和嵌入任务来同时处理两者，该模型在大规模文本嵌入基准（MTEB）上取得了最佳结果，并在多个生成性任务中超越同类模型。通过进一步扩展，我们的模型在仍属于最佳嵌入模型的同时，优于我们尝试过的所有公开生成性语言模型。此外，我们发现 GRIT 能够实现生成性或嵌入数据的训练，并且无需性能损失。