With the methodological support of probing (or diagnostic classification), recent studies have demonstrated that Transformers encode syntactic and semantic information to some extent. Following this line of research, this paper aims at taking semantic probing to an abstraction extreme with the goal of answering the following research question: can contemporary Transformer-based models reflect an underlying Foundational Ontology? To this end, we present a systematic Foundational Ontology (FO) probing methodology to investigate whether Transformers-based models encode abstract semantic information. Following different pre-training and fine-tuning regimes, we present an extensive evaluation of a diverse set of large-scale language models over three distinct and complementary FO tagging experiments. Specifically, we present and discuss the following conclusions: (1) The probing results indicate that Transformer-based models incidentally encode information related to Foundational Ontologies during the pre-training pro-cess; (2) Robust FO taggers (accuracy of 90 percent)can be efficiently built leveraging on this knowledge.

本文采用语言学的研究方法探查 Transformers 语言模型是否能反映 Foundational Ontology 等抽象语义信息，并给出丰富的实验证明：在预训练过程中，Transformer 基础模型可以编码与 FO 相关的信息，基于此，可以高效地构建鲁棒的 FO 识别器（准确度达到 90%）

Transformer是否编码了基础本体论？探究自然语言中的抽象类