Large language models (LLMs) face issues in handling factual knowledge,
making it vital to evaluate their true ability to understand facts. In this
study, we introduce knowledge probing frameworks, BELIEF(-ICL), to evaluate the
knowledge understanding ability of not only encoder-based PLMs but also
decoder-based PLMs from diverse perspectives. BELIEFs utilize a multi-prompt
dataset to evaluate PLM's accuracy, consistency, and reliability in factual
knowledge understanding. To provide a more reliable evaluation with BELIEFs, we
semi-automatically create MyriadLAMA, which has more diverse prompts than
existing datasets. We validate the effectiveness of BELIEFs in correctly and
comprehensively evaluating PLM's factual understanding ability through
extensive evaluations. We further investigate key factors in learning facts in
LLMs, and reveal the limitation of the prompt-based knowledge probing. The
dataset is anonymously publicized.

大型语言模型 (LLMs) 面临处理事实知识的问题，本研究通过知识探测框架 BELIEF (-ICL)，从多个角度评估编码器型和解码器型大型语言模型对事实知识的理解能力，并利用多样的提示数据集 MyriadLAMA 进行可靠的评估，揭示了大型语言模型在学习事实方面的关键因素和基于提示的知识探测的局限性。