Large language models (LLMs) have recently revolutionized automated text
understanding and generation. The performance of these models relies on the
high number of parameters of the underlying neural architectures, which allows
LLMs to memorize part of the vast quantity of data seen during the training.
This paper investigates whether and to what extent general-purpose pre-trained
LLMs have memorized information from known ontologies. Our results show that
LLMs partially know ontologies: they can, and do indeed, memorize concepts from
ontologies mentioned in the text, but the level of memorization of their
concepts seems to vary proportionally to their popularity on the Web, the
primary source of their training material. We additionally propose new metrics
to estimate the degree of memorization of ontological information in LLMs by
measuring the consistency of the output produced across different prompt
repetitions, query languages, and degrees of determinism.

大型语言模型能够部分地记忆本体论概念，并且记忆程度与概念在网络上的普及程度成正比。本研究还提出了新的度量方法，通过测量在不同提示重复、查询语言和确定性程度下产生的输出的一致性来估计语言模型中本体论信息的记忆程度。