Recent approaches to improving the extraction of text embeddings from
autoregressive large language models (LLMs) have largely focused on
improvements to data, backbone pretrained language models, or improving
task-differentiation via instructions. In this work, we address an
architectural limitation of autoregressive models: token embeddings cannot
contain information from tokens that appear later in the input. To address this
limitation, we propose a simple approach, "echo embeddings," in which we repeat
the input twice in context and extract embeddings from the second occurrence.
We show that echo embeddings of early tokens can encode information about later
tokens, allowing us to maximally leverage high-quality LLMs for embeddings. On
the MTEB leaderboard, echo embeddings improve over classical embeddings by over
9% zero-shot and by around 0.7% when fine-tuned. Echo embeddings with a
Mistral-7B model achieve state-of-the-art compared to prior open source models
that do not leverage synthetic fine-tuning data.

提供了一种解决自回归模型的架构限制的简单方法，即 “回声嵌入”，通过在上下文中重复输入并从第二次出现中提取嵌入，使得提前的标记嵌入能够包含后面的标记的信息，从而最大程度地利用高质量 LLMs 进行嵌入。在 MTEB 榜单上，回声嵌入在零 - shot 情况下改进了超过 9％，细调时改进了约 0.7％。用 Mistral-7B 模型的回声嵌入相较于之前没有利用合成调优数据的开源模型达到了最新技术水平。