Existing sentence textual similarity benchmark datasets only use a single
number to summarize how similar the sentence encoder's decision is to humans'.
However, it is unclear what kind of sentence pairs a sentence encoder (SE)
would consider similar. Moreover, existing SE benchmarks mainly consider
sentence pairs with low lexical overlap, so it is unclear how the SEs behave
when two sentences have high lexical overlap. We introduce a high-quality SE
diagnostic dataset, HEROS. HEROS is constructed by transforming an original
sentence into a new sentence based on certain rules to form a \textit{minimal
pair}, and the minimal pair has high lexical overlaps. The rules include
replacing a word with a synonym, an antonym, a typo, a random word, and
converting the original sentence into its negation. Different rules yield
different subsets of HEROS. By systematically comparing the performance of over
60 supervised and unsupervised SEs on HEROS, we reveal that most unsupervised
sentence encoders are insensitive to negation. We find the datasets used to
train the SE are the main determinants of what kind of sentence pairs an SE
considers similar. We also show that even if two SEs have similar performance
on STS benchmarks, they can have very different behavior on HEROS. Our result
reveals the blind spot of traditional STS benchmarks when evaluating SEs.

通过构建一个高质量的 SE 诊断数据集 HEROS，在系统比较了 60 多个监督和无监督的 SE 在 HEROS 上的表现后，我们发现大多数无监督的句子编码器对否定不敏感，并且发现用于训练 SE 的数据集是决定 SE 认为哪些句子对相似的主要因素，同时揭示了传统 STS 基准测试在评估 SE 时的盲点。