Generative Language Models (GLMs) have demonstrated capabilities to store
factual knowledge and answer queries efficiently. Given varying prompts, does a
GLM consistently generate factually correct answers? In this paper, we
introduce a statistical knowledge assessment framework guided by latent
variables and the KaRR metric, which quantifies a model's knowledge by
computing its continuous probability across diverse text forms. We conduct a
comprehensive comparison of knowledge across 14 GLMs using our framework,
including LLaMA, Alpaca, OPT, and others. Our statistical knowledge assessment
encompasses 600 relation types and exhibits a strong correlation (0.43
Kendall's $\tau$) with human evaluation. Our findings reveal that the knowledge
in GLMs with the same backbone architecture adheres to the scaling law, and
that tuning on instruction-following data may compromise the model's ability to
generate factually correct text consistently.

采用潜在变量和 KaRR 指标引导的统计知识评估框架对 14 种生成语言模型 (GLMs) 进行全面比较；该研究表明，具有相同骨干结构的 GLMs 的知识符合缩放定律，并且在指令遵循数据上的调整可能会危及模型始终生成正确的文本的能力。