Before deploying a language model (LM) within a given domain, it is important
to measure its tendency to generate factually incorrect information in that
domain. Existing factual generation evaluation methods focus on facts sampled
from the LM itself, and thus do not control the set of evaluated facts and
might under-represent rare and unlikely facts. We propose FACTOR: Factual
Assessment via Corpus TransfORmation, a scalable approach for evaluating LM
factuality. FACTOR automatically transforms a factual corpus of interest into a
benchmark evaluating an LM's propensity to generate true facts from the corpus
vs. similar but incorrect statements. We use our framework to create two
benchmarks: Wiki-FACTOR and News-FACTOR. We show that: (i) our benchmark scores
increase with model size and improve when the LM is augmented with retrieval;
(ii) benchmark score correlates with perplexity, but the two metrics do not
always agree on model ranking; and (iii) when perplexity and benchmark score
disagree, the latter better reflects factuality in open-ended generation, as
measured by human annotators. We make our data and code publicly available in
this https URL

本文通过提出 FACTOR 方法，使用自动转换真实语料库的方式评估语言模型的事实推理能力，并通过 Wiki-FACTOR 和 News-FACTOR 两个基准测试数据集测试方法的有效性。