In this paper, we adopted a retrospective approach to examine and compare
five existing popular sentence encoders, i.e., Sentence-BERT, Universal
Sentence Encoder (USE), LASER, InferSent, and Doc2vec, in terms of their
performance on downstream tasks versus their capability to capture basic
semantic properties. Initially, we evaluated all five sentence encoders on the
popular SentEval benchmark and found that multiple sentence encoders perform
quite well on a variety of popular downstream tasks. However, being unable to
find a single winner in all cases, we designed further experiments to gain a
deeper understanding of their behavior. Specifically, we proposed four semantic
evaluation criteria, i.e., Paraphrasing, Synonym Replacement, Antonym
Replacement, and Sentence Jumbling, and evaluated the same five sentence
encoders using these criteria. We found that the Sentence-Bert and USE models
pass the paraphrasing criterion, with SBERT being the superior between the two.
LASER dominates in the case of the synonym replacement criterion.
Interestingly, all the sentence encoders failed the antonym replacement and
jumbling criteria. These results suggest that although these popular sentence
encoders perform quite well on the SentEval benchmark, they still struggle to
capture some basic semantic properties, thus, posing a daunting dilemma in NLP
research.

在本研究中，我们采用回顾性方法，对比了 5 种现有的句子编码器，即 Sentence-BERT、Universal Sentence Encoder (USE)、LASER、InferSent 和 Doc2vec，在下游任务表现和捕获基本语义特征能力方面的表现。我们评估了这五种句子编码器在受欢迎的 SentEval 基准测试上的表现，发现多个句子编码器在各种受欢迎的下游任务上表现良好。然而，在所有情况下都没有找到一个单一的优胜者，因此，我们设计了进一步的实验来更深入地了解它们的行为。我们提出了四个语义评估标准：复述、同义词替换、反义词替换和句子混乱，并使用这些标准评估了同样的五种句子编码器。我们发现，Sentence-BERT 和 USE 模型通过了复述标准，其中 SBERT 在两者之间更为优越。在同义词替换标准方面，LASER 表现最佳。有趣的是，所有句子编码器都未通过反义词替换和句子混乱的标准。这些结果表明，尽管这些受欢迎的句子编码器在 SentEval 基准测试上表现良好，但它们仍然难以捕捉一些基本的语义特征，因此，在自然语言处理研究中面临严峻的困境。