This paper examines the encoding of analogy in large-scale pretrained
language models, such as BERT and GPT-2. Existing analogy datasets typically
focus on a limited set of analogical relations, with a high similarity of the
two domains between which the analogy holds. As a more realistic setup, we
introduce the Scientific and Creative Analogy dataset (SCAN), a novel analogy
dataset containing systematic mappings of multiple attributes and relational
structures across dissimilar domains. Using this dataset, we test the
analogical reasoning capabilities of several widely-used pretrained language
models (LMs). We find that state-of-the-art LMs achieve low performance on
these complex analogy tasks, highlighting the challenges still posed by analogy
understanding.

本文研究了大规模预训练语言模型（如 BERT 和 GPT-2）中类比的编码。我们介绍了一个新的类比数据集（SCAN），包含跨不同领域的多个属性和关系结构的系统映射，测试了几种广泛使用的预训练语言模型（LMs）的类比推理能力。然后发现，最先进的 LMs 在这些复杂的类比任务中表现不佳，突出了类比理解仍然存在的挑战。