Pretrained language models (LMs) are susceptible to generate text with
nonfactual information. In this work, we measure and improve the factual
accuracy of large-scale LMs for open-ended text generation. We design the
FactualityPrompts test set and metrics to measure the factuality of LM
generations. Based on that, we study the factual accuracy of LMs with parameter
sizes ranging from 126M to 530B. Interestingly, we find that larger LMs are
more factual than smaller ones, although a previous study suggests that larger
LMs can be less truthful in terms of misconceptions. In addition, popular
sampling algorithms (e.g., top-p) in open-ended text generation can harm the
factuality due to the ''uniform randomness'' introduced at every sampling step.
We propose the factual-nucleus sampling algorithm that dynamically adapts the
randomness to improve the factuality of generation while maintaining quality.
Furthermore, we analyze the inefficiencies of the standard training method in
learning correct associations between entities from factual text corpus (e.g.,
Wikipedia). We propose a factuality-enhanced training method that uses
TopicPrefix for better awareness of facts and sentence completion as the
training objective, which can vastly reduce the factual errors. We release our
code and FactualityPrompts benchmark at:
this https URL.

本研究针对大规模预训练的语言模型，设计测试集和度量标准以提高生成文本的事实准确性，提出了基于主题前缀和句子补全的事实增强训练方法，并提出了更适合提高准确性的采样算法。