Learning meaningful word embeddings is key to training a robust language
model. The recent rise of Large Language Models (LLMs) has provided us with
many new word/sentence/document embedding models. Although LLMs have shown
remarkable advancement in various NLP tasks, it is still unclear whether the
performance improvement is merely because of scale or whether underlying
embeddings they produce significantly differ from classical encoding models
like Sentence-BERT (SBERT) or Universal Sentence Encoder (USE). This paper
systematically investigates this issue by comparing classical word embedding
techniques against LLM-based word embeddings in terms of their latent vector
semantics. Our results show that LLMs tend to cluster semantically related
words more tightly than classical models. LLMs also yield higher average
accuracy on the Bigger Analogy Test Set (BATS) over classical methods. Finally,
some LLMs tend to produce word embeddings similar to SBERT, a relatively
lighter classical model.

通过比较经典词嵌入技术与大型语言模型的词嵌入之间的潜在向量语义，系统地调查了大型语言模型是否在表现上与经典编码模型存在显著差异。结果显示，大型语言模型往往比经典模型更紧密地聚集语义相关的词，并在 Bigger Analogy Test Set (BATS) 上取得更高的平均准确率。此外，一些大型语言模型的词嵌入与相对较轻的句子级 BERT (SBERT) 模型相似。