Text embedding models have significantly contributed to advancements in natural language processing by adeptly capturing semantic properties of textual data. However, the ability of these models to generalize across a wide range of syntactic contexts remains under-explored. In this paper, we first develop an evaluation set, named \textbf{SR}, to scrutinize the capability for syntax understanding of text embedding models from two crucial syntactic aspects: Structural heuristics, and Relational understanding among concepts, as revealed by the performance gaps in previous studies. Our findings reveal that existing text embedding models have not sufficiently addressed these syntactic understanding challenges, and such ineffectiveness becomes even more apparent when evaluated against existing benchmark datasets. Furthermore, we conduct rigorous analysis to unearth factors that lead to such limitations and examine why previous evaluations fail to detect such ineffectiveness. Lastly, we propose strategies to augment the generalization ability of text embedding models in diverse syntactic scenarios. This study serves to highlight the hurdles associated with syntactic generalization and provides pragmatic guidance for boosting model performance across varied syntactic contexts.

本研究通过开发评估集SR来审查文本嵌入模型在句法理解能力方面的表现，发现现有的文本嵌入模型在面对句法理解挑战时效果不佳，尤其在与现有基准数据集的对比中更加明显。同时，通过深入分析揭示了导致这些局限性的因素，并提出了增强文本嵌入模型在不同句法场景中泛化能力的策略，为强化模型在各类句法上下文中的性能提供了实用指南。

文本嵌入模型对语法的理解程度如何？