Text embeddings from large language models (LLMs) have achieved excellent
results in tasks such as information retrieval, semantic textual similarity,
etc. In this work, we show an interesting finding: when feeding a text into the
embedding LLMs, the obtained text embedding will be able to be aligned with the
key tokens in the input text. We first fully analyze this phenomenon on eight
embedding LLMs and show that this phenomenon is universal and is not affected
by model architecture, training strategy, and embedding method. With a deeper
analysis, we then find that the main change in embedding space between the
embedding LLMs and their original generative LLMs is in the first principal
component. By adjusting the first principal component, we can align text
embedding with the key tokens. Finally, we give several examples to demonstrate
the vast application potential of this finding: (1) we propose a simple and
practical sparse retrieval method based on the aligned tokens, which can
achieve 80\% of the dense retrieval effect of the same model while reducing the
computation significantly; (2) we show that our findings provide a fresh
perspective to help understand fuzzy concepts (e.g., semantic relatedness vs.
semantic similarity) and emerging technologies (e.g., instruction-following
embedding) in this field.

通过分析长序列模型的嵌入数据，我们发现其文本嵌入与关键标记是可以对齐的，并进一步通过调整首个主要成分，实现文本嵌入与关键标记的对齐。这一发现拥有广泛应用潜力，可用于稀疏检索方法的改进和模糊概念的理解。

一段文字胜过数个令牌：LLM 的文字嵌入与关键令牌密切对齐

A Text is Worth Several Tokens: Text Embedding from LLMs Secretly Aligns  Well with The Key Tokens

Ensembling multiple models has always been an effective approach to push the
limits of existing performance and is widely used in classification tasks by
simply averaging the classification probability vectors from multiple
classifiers to achieve better accuracy. However, in the thriving open-source
Large Language Model (LLM) community, ensembling methods are rare and typically
limited to ensembling the full-text outputs of LLMs, such as selecting the best
output using a ranker, which leads to underutilization of token-level
probability information. In this paper, we treat the Generation of each token
by LLMs as a Classification (GaC) for ensembling. This approach fully exploits
the probability information at each generation step and better prevents LLMs
from producing early incorrect tokens that lead to snowballing errors. In
experiments, we ensemble state-of-the-art LLMs on several benchmarks, including
exams, mathematics and reasoning, and observe that our method breaks the
existing community performance ceiling. Furthermore, we observed that most of
the tokens in the answer are simple and do not affect the correctness of the
final answer. Therefore, we also experimented with ensembling only key tokens,
and the results showed better performance with lower latency across benchmarks.

利用生成式语言模型（GaC），将多个大型语言模型（LLM）集成在一起，在多项基准测试中突破了现有的性能限制；此外，仅集成关键标记也表现出更好的性能和更低的延迟。