Despite the subjective nature of semantic textual similarity (STS) and
pervasive disagreements in STS annotation, existing benchmarks have used
averaged human ratings as the gold standard. Averaging masks the true
distribution of human opinions on examples of low agreement, and prevents
models from capturing the semantic vagueness that the individual ratings
represent. In this work, we introduce USTS, the first Uncertainty-aware STS
dataset with ~15,000 Chinese sentence pairs and 150,000 labels, to study
collective human opinions in STS. Analysis reveals that neither a scalar nor a
single Gaussian fits a set of observed judgements adequately. We further show
that current STS models cannot capture the variance caused by human
disagreement on individual instances, but rather reflect the predictive
confidence over the aggregate dataset.

本研究提出了第一个拥有约 15,000 个中文句子对和 150,000 个标签的不确定性感知语义文本相似度（STS）数据集（USTS），旨在研究 STS 中的集体人类观点。通过分析可以发现，一个标量或一个高斯分布无法充分地适应观测到的判定集合。我们进一步展示了当前的 STS 模型无法捕捉到由个别实例上的人类意见分歧引起的方差，而是反映了对整体数据集的预测置信度。