This paper investigates the efficacy of RWKV, a novel language model architecture known for its linear attention mechanism, for generating sentence embeddings in a zero-shot setting. I conduct a layer-wise analysis to evaluate the semantic similarity captured by embeddings from different hidden layers of a pre-trained RWKV model. The performance is assessed on the Microsoft Research Paraphrase Corpus (MRPC) dataset using Spearman correlation and compared against a GloVe-based baseline. My results indicate that while RWKV embeddings capture some semantic relatedness, they underperform compared to the GloVe baseline in terms of Spearman correlation. I also analyze the inference time and GPU memory usage, highlighting the computational trade-offs associated with RWKV embeddings. The findings suggest that while RWKV offers potential advantages in terms of linear scaling, its zero-shot sentence embedding quality for semantic similarity tasks requires further investigation and potential task-specific fine-tuning to match or exceed simpler baselines.

本研究关注RWKV语言模型在零样本条件下生成句子嵌入的效果，填补了RWKV在语义相似性任务上的研究空白。通过逐层分析预训练RWKV模型的不同隐藏层嵌入，评估了其语义相似性，并与基于GloVe的基线进行了比较。结果显示，虽然RWKV在捕捉一些语义相关性方面表现出潜力，但整体性能未能超越GloVe基线，且在计算效率上存在权衡，提示未来需进一步探讨其在特定任务中的调优潜力。

探讨RWKV在句子嵌入中的应用：逐层分析与基线比较语义相似性