Semantic textual similarity (STS) has been a cornerstone task in NLP that
measures the degree of similarity between a pair of sentences, with
applications in information retrieval, question answering, and embedding
methods. However, it is an inherently ambiguous task, with the sentence
similarity depending on the specific aspect of interest. We resolve this
ambiguity by proposing a novel task called conditional STS (C-STS) which
measures similarity conditioned on an aspect elucidated in natural language
(hereon, condition). As an example, the similarity between the sentences "The
NBA player shoots a three-pointer." and "A man throws a tennis ball into the
air to serve." is higher for the condition "The motion of the ball." (both
upward) and lower for "The size of the ball." (one large and one small).
C-STS's advantages are two-fold: (1) it reduces the subjectivity and ambiguity
of STS, and (2) enables fine-grained similarity evaluation using diverse
conditions. C-STS contains almost 20,000 instances from diverse domains and we
evaluate several state-of-the-art models to demonstrate that even the most
performant fine-tuning and in-context learning models (GPT-4, Flan, SimCSE)
find it challenging, with Spearman correlation scores of <50. We encourage the
community to evaluate their models on C-STS to provide a more holistic view of
semantic similarity and natural language understanding.

通过提出一种新的条件语义文本相似度任务 (conditional STS，C-STS)，旨在找到一个针对性的自然语言条件方面来测量句子之间的相似度，可以减少 STS 的主观性和模糊性并利用多样的条件实现细粒度的相似度评估，并使用各种最先进的模型证明了 C-STS 的挑战性。