We present Gecko, a compact and versatile text embedding model. Gecko
achieves strong retrieval performance by leveraging a key idea: distilling
knowledge from large language models (LLMs) into a retriever. Our two-step
distillation process begins with generating diverse, synthetic paired data
using an LLM. Next, we further refine the data quality by retrieving a set of
candidate passages for each query, and relabeling the positive and hard
negative passages using the same LLM. The effectiveness of our approach is
demonstrated by the compactness of the Gecko. On the Massive Text Embedding
Benchmark (MTEB), Gecko with 256 embedding dimensions outperforms all existing
entries with 768 embedding size. Gecko with 768 embedding dimensions achieves
an average score of 66.31, competing with 7x larger models and 5x higher
dimensional embeddings.

我们提出了一种紧凑而多功能的文本嵌入模型 Gecko，其通过利用大语言模型（LLMs）将知识从 LLMs 中提炼到检索器中来实现强大的检索性能。

Gecko：从大型语言模型中提取的多功能文本嵌入

Gecko: Versatile Text Embeddings Distilled from Large Language Models

Conversational multi-doc question answering aims to answer specific questions
based on the retrieved documents as well as the contextual conversations. In
this paper, we introduce our winning approach for the "Conversational Multi-Doc
QA" challenge in WSDM Cup 2024, which exploits the superior natural language
understanding and generation capability of Large Language Models (LLMs). We
first adapt LLMs to the task, then devise a hybrid training strategy to make
the most of in-domain unlabeled data. Moreover, an advanced text embedding
model is adopted to filter out potentially irrelevant documents and several
approaches are designed and compared for the model ensemble. Equipped with all
these techniques, our solution finally ranked 1st place in WSDM Cup 2024,
surpassing its rivals to a large extent. The source codes have been released at
this https URL

我们介绍了我们在 WSDM Cup 2024 的 “Conversational Multi-Doc QA” 挑战中的获胜方法，利用了大型语言模型（LLM）的优越自然语言理解和生成能力，通过将 LLMs 调整到任务需求，制定了混合训练策略来充分利用领域内无标注数据，并采用先进的文本嵌入模型来过滤可能不相关的文档，并设计和比较多种模型集成的方法，配备了所有这些技术，我们的解决方案最终在 WSDM Cup 2024 中获得了第一名，大大超过了竞争对手。