Many applications require grouping instances contained in diverse document
datasets into classes. Most widely used methods do not employ deep learning and
do not exploit the inherently multimodal nature of documents. Notably, record
linkage is typically conceptualized as a string-matching problem. This study
develops CLIPPINGS, (Contrastively Linking Pooled Pre-trained Embeddings), a
multimodal framework for record linkage. CLIPPINGS employs end-to-end training
of symmetric vision and language bi-encoders, aligned through contrastive
language-image pre-training, to learn a metric space where the pooled
image-text representation for a given instance is close to representations in
the same class and distant from representations in different classes. At
inference time, instances can be linked by retrieving their nearest neighbor
from an offline exemplar embedding index or by clustering their
representations. The study examines two challenging applications: constructing
comprehensive supply chains for mid-20th century Japan through linking firm
level financial records - with each firm name represented by its crop in the
document image and the corresponding OCR - and detecting which image-caption
pairs in a massive corpus of historical U.S. newspapers came from the same
underlying photo wire source. CLIPPINGS outperforms widely used string matching
methods by a wide margin and also outperforms unimodal methods. Moreover, a
purely self-supervised model trained on only image-OCR pairs also outperforms
popular string-matching methods without requiring any labels.

该研究开发了一个名为 CLIPPINGS 的多模态框架，它利用端到端训练，对称视觉和语言双编码器，并通过对比语言 - 图像预训练来将它们对齐，以学习一个度量空间，在这个空间中，给定实例的汇总图像 - 文本表示对于同一类别的表示非常接近，对于不同类别的表示则很遥远。该框架在两个应用方面的表现都超过了广泛使用的字符串匹配方法，并且在不需要任何标签的情况下，仅基于图像 - OCR 对进行自我监督训练的纯自监督模型也比受欢迎的字符串匹配方法表现更好。