Exploring and quantifying semantic relatedness is central to representing
language. It holds significant implications across various NLP tasks, including
offering insights into the capabilities and performance of Large Language
Models (LLMs). While earlier NLP research primarily focused on semantic
similarity, often within the English language context, we instead investigate
the broader phenomenon of semantic relatedness. In this paper, we present
SemRel, a new semantic relatedness dataset collection annotated by native
speakers across 14 languages:Afrikaans, Algerian Arabic, Amharic, English,
Hausa, Hindi, Indonesian, Kinyarwanda, Marathi, Moroccan Arabic, Modern
Standard Arabic, Punjabi, Spanish, and Telugu. These languages originate from
five distinct language families and are predominantly spoken in Africa and Asia
-- regions characterised by a relatively limited availability of NLP resources.
Each instance in the SemRel datasets is a sentence pair associated with a score
that represents the degree of semantic textual relatedness between the two
sentences. The scores are obtained using a comparative annotation framework. We
describe the data collection and annotation processes, related challenges when
building the datasets, and their impact and utility in NLP. We further report
experiments for each language and across the different languages.

探索和量化语义相关性对于表示语言至关重要，它对各种 NLP 任务具有重要意义，包括提供关于大型语言模型能力和性能的见解。本文介绍了 SemRel，这是一个由母语为 14 种语言的本土人士进行注释的新的语义相关性数据集收集项目。每个实例都是一个句子对，并与表示两个句子之间语义文本相关性程度的分数相关联。分数是通过比较性注释框架获得的。我们描述了数据收集和注释过程、构建数据集时的相关挑战以及它们在 NLP 中的影响和实用性。我们还报告了对每种语言以及不同语言之间进行的实验。