Semantic textual similarity (STS) is a fundamental NLP task that measures the
semantic similarity between a pair of sentences. In order to reduce the
inherent ambiguity posed from the sentences, a recent work called Conditional
STS (C-STS) has been proposed to measure the sentences' similarity conditioned
on a certain aspect. Despite the popularity of C-STS, we find that the current
C-STS dataset suffers from various issues that could impede proper evaluation
on this task. In this paper, we reannotate the C-STS validation set and observe
an annotator discrepancy on 55% of the instances resulting from the annotation
errors in the original label, ill-defined conditions, and the lack of clarity
in the task definition. After a thorough dataset analysis, we improve the C-STS
task by leveraging the models' capability to understand the conditions under a
QA task setting. With the generated answers, we present an automatic error
identification pipeline that is able to identify annotation errors from the
C-STS data with over 80% F1 score. We also propose a new method that largely
improves the performance over baselines on the C-STS data by training the
models with the answers. Finally we discuss the conditionality annotation based
on the typed-feature structure (TFS) of entity types. We show in examples that
the TFS is able to provide a linguistic foundation for constructing C-STS data
with new conditions.

通过重新注释 C-STS 验证集，并经过全面的数据集分析，我们改进了 C-STS 任务，利用模型理解 QA 任务设置中的条件，从而生成答案，并提出了一种自动错误识别流水线，能够以 80% 以上的 F1 得分识别 C-STS 数据中的注释错误。此外，我们还提出了一种新方法，通过将答案与模型进行训练，大大提高了 C-STS 数据上的性能。最后，我们讨论了基于实体类型的条件性注释方法，并展示了实例中，基于类型 - 特征结构 (TFS) 的方法能够为构建具有新条件的 C-STS 数据提供语言基础。

基于语言条件的语义文本相似度

Linguistically Conditioned Semantic Textual Similarity

Semantic textual similarity (STS) has been a cornerstone task in NLP that
measures the degree of similarity between a pair of sentences, with
applications in information retrieval, question answering, and embedding
methods. However, it is an inherently ambiguous task, with the sentence
similarity depending on the specific aspect of interest. We resolve this
ambiguity by proposing a novel task called conditional STS (C-STS) which
measures similarity conditioned on an aspect elucidated in natural language
(hereon, condition). As an example, the similarity between the sentences "The
NBA player shoots a three-pointer." and "A man throws a tennis ball into the
air to serve." is higher for the condition "The motion of the ball." (both
upward) and lower for "The size of the ball." (one large and one small).
C-STS's advantages are two-fold: (1) it reduces the subjectivity and ambiguity
of STS, and (2) enables fine-grained similarity evaluation using diverse
conditions. C-STS contains almost 20,000 instances from diverse domains and we
evaluate several state-of-the-art models to demonstrate that even the most
performant fine-tuning and in-context learning models (GPT-4, Flan, SimCSE)
find it challenging, with Spearman correlation scores of <50. We encourage the
community to evaluate their models on C-STS to provide a more holistic view of
semantic similarity and natural language understanding.

通过提出一种新的条件语义文本相似度任务 (conditional STS，C-STS)，旨在找到一个针对性的自然语言条件方面来测量句子之间的相似度，可以减少 STS 的主观性和模糊性并利用多样的条件实现细粒度的相似度评估，并使用各种最先进的模型证明了 C-STS 的挑战性。