Conceptual knowledge is fundamental to human cognition and knowledge bases.
However, existing knowledge probing works only focus on evaluating factual
knowledge of pre-trained language models (PLMs) and ignore conceptual
knowledge. Since conceptual knowledge often appears as implicit commonsense
behind texts, designing probes for conceptual knowledge is hard. Inspired by
knowledge representation schemata, we comprehensively evaluate conceptual
knowledge of PLMs by designing three tasks to probe whether PLMs organize
entities by conceptual similarities, learn conceptual properties, and
conceptualize entities in contexts, respectively. For the tasks, we collect and
annotate 24k data instances covering 393 concepts, which is COPEN, a COnceptual
knowledge Probing bENchmark. Extensive experiments on different sizes and types
of PLMs show that existing PLMs systematically lack conceptual knowledge and
suffer from various spurious correlations. We believe this is a critical
bottleneck for realizing human-like cognition in PLMs. COPEN and our codes are
publicly released at this https URL.

本研究提出了 COPEN，作为探查预训练语言模型概念知识的评估基准，并通过三项任务对 PLMs 的概念知识进行综合评估，结果表明现有的 PLMs 系统性缺乏概念知识并受到各种虚假相关性的影响。

COPEN：探测预训练语言模型中的概念知识

COPEN: Probing Conceptual Knowledge in Pre-trained Language Models

When humans read or listen, they make implicit commonsense inferences that
frame their understanding of what happened and why. As a step toward AI systems
that can build similar mental models, we introduce GLUCOSE, a large-scale
dataset of implicit commonsense causal knowledge, encoded as causal
mini-theories about the world, each grounded in a narrative context. To
construct GLUCOSE, we drew on cognitive psychology to identify ten dimensions
of causal explanation, focusing on events, states, motivations, and emotions.
Each GLUCOSE entry includes a story-specific causal statement paired with an
inference rule generalized from the statement. This paper details two concrete
contributions. First, we present our platform for effectively crowdsourcing
GLUCOSE data at scale, which uses semi-structured templates to elicit causal
explanations. Using this platform, we collected a total of ~670K specific
statements and general rules that capture implicit commonsense knowledge about
everyday situations. Second, we show that existing knowledge resources and
pretrained language models do not include or readily predict GLUCOSE's rich
inferential content. However, when state-of-the-art neural models are trained
on this knowledge, they can start to make commonsense inferences on unseen
stories that match humans' mental models.

本文介绍了一个大规模的神经启发式表示系统 GLUCOSE，以建立 AI 的基于常识的推理模型，采用半结构化模板进行数据收集，数据中心包括特定故事的因果陈述和推广自该陈述的演绎规则。文章还讨论了 GLUCOSE 的预训练模型可以启发神经模型进行一般推理的做法。