A diversity of tasks use language models trained on semantic similarity data.
While there are a variety of datasets that capture semantic similarity, they
are either constructed from modern web data or are relatively small datasets
created in the past decade by human annotators. This study utilizes a novel
source, newly digitized articles from off-copyright, local U.S. newspapers, to
assemble a massive-scale semantic similarity dataset spanning 70 years from
1920 to 1989 and containing nearly 400M positive semantic similarity pairs.
Historically, around half of articles in U.S. local newspapers came from
newswires like the Associated Press. While local papers reproduced articles
from the newswire, they wrote their own headlines, which form abstractive
summaries of the associated articles. We associate articles and their headlines
by exploiting document layouts and language understanding. We then use deep
neural methods to detect which articles are from the same underlying source, in
the presence of substantial noise and abridgement. The headlines of reproduced
articles form positive semantic similarity pairs. The resulting publicly
available HEADLINES dataset is significantly larger than most existing semantic
similarity datasets and covers a much longer span of time. It will facilitate
the application of contrastively trained semantic similarity models to a
variety of tasks, including the study of semantic change across space and time.

该研究利用来自当地美国报纸的新数字化文章创建了一个巨大的语义相似性数据集，并利用深度神经方法检测了这些文章中的正面语义相似性对。该语义相似性数据集跨足了 70 年，包含近 400M 个正面语义相似性对，随着时间跨度的增加，将有助于将对比训练的语义相似性模型应用于各种任务。