The significant advancements of Large Language Models (LLMs) in generative tasks have led to a growing body of work exploring LLM-based embedding models. While these models, employing different pooling and attention strategies, have achieved state-of-the-art performance on public embedding benchmarks, questions still arise about what constitutes an effective design for LLM-based embedding models. However, these models are often trained on different datasets, using different LLM base models or training settings. Moreover, evaluations on public embedding benchmarks often fail to report statistical significance, making it difficult to determine which designs truly contribute to final performance. This complicates the process for practitioners seeking optimal training recipes for LLM-based embedding models. In this study, we conduct a large-scale experiment by training a series of LLM-based embedding models using the same training data and base model but differing in their pooling and attention strategies. The results show that there is no one-size-fits-all solution: while bidirectional attention and an additional trainable pooling layer outperform in text similarity and information retrieval tasks, they do not significantly surpass simpler designs like EOS-last token pooling and default causal attention in clustering and classification tasks. Furthermore, we propose a new pooling strategy, Multi-Layers Trainable Pooling, which transforms the outputs of all hidden layers, rather than just the last layer, using a cross-attention network. This method proves to be statistically superior in text similarity and retrieval tasks compared to existing pooling methods. Overall, this paper sheds light on effective training strategies for LLM-based embedding models.

本研究解决了基于大语言模型（LLM）的嵌入模型设计的有效性问题，探索不同池化和注意力策略的影响。通过对一系列使用相同训练数据和基础模型的嵌入模型进行大规模实验，发现虽然双向注意力和可训练池化层在文本相似性和信息检索任务中表现优异，但在聚类和分类任务中并不显著优于简单设计。此外，提出的新型多层可训练池化策略在文本相似性和检索任务中展现出统计上的优势，提供了有效的训练策略洞察。 

池化与注意力：基于LLM的嵌入模型的有效设计是什么？