Text embeddings from large language models (LLMs) have achieved excellent
results in tasks such as information retrieval, semantic textual similarity,
etc. In this work, we show an interesting finding: when feeding a text into the
embedding LLMs, the obtained text embedding will be able to be aligned with the
key tokens in the input text. We first fully analyze this phenomenon on eight
embedding LLMs and show that this phenomenon is universal and is not affected
by model architecture, training strategy, and embedding method. With a deeper
analysis, we then find that the main change in embedding space between the
embedding LLMs and their original generative LLMs is in the first principal
component. By adjusting the first principal component, we can align text
embedding with the key tokens. Finally, we give several examples to demonstrate
the vast application potential of this finding: (1) we propose a simple and
practical sparse retrieval method based on the aligned tokens, which can
achieve 80\% of the dense retrieval effect of the same model while reducing the
computation significantly; (2) we show that our findings provide a fresh
perspective to help understand fuzzy concepts (e.g., semantic relatedness vs.
semantic similarity) and emerging technologies (e.g., instruction-following
embedding) in this field.

通过分析长序列模型的嵌入数据，我们发现其文本嵌入与关键标记是可以对齐的，并进一步通过调整首个主要成分，实现文本嵌入与关键标记的对齐。这一发现拥有广泛应用潜力，可用于稀疏检索方法的改进和模糊概念的理解。