LLM hallucination, i.e. generating factually incorrect yet seemingly
convincing answers, is currently a major threat to the trustworthiness and
reliability of LLMs. The first step towards solving this complicated problem is
to measure it. However, existing hallucination metrics require to have a
benchmark dataset with gold-standard answers, i.e. "best" or "correct" answers
written by humans. Such requirement makes hallucination measurement costly and
prone to human errors. In this work, we propose Factualness Evaluations via
Weighting LLMs (FEWL), the first hallucination metric that is specifically
designed for the scenario when gold-standard answers are absent. FEWL leverages
the answers from off-the-shelf LLMs that serve as a proxy of gold-standard
answers. The key challenge is how to quantify the expertise of reference LLMs
resourcefully. We show FEWL has certain theoretical guarantees and demonstrate
empirically it gives more accurate hallucination measures than naively using
reference LLMs. We also show how to leverage FEWL to reduce hallucination
through both in-context learning and supervised finetuning. Last, we build a
large-scale benchmark dataset to facilitate LLM hallucination research.

通过权重 LLMs 的事实性评估（FEWL），我们提出了第一个专门针对没有黄金标准答案的情况的幻觉度量标准，该度量标准借助现成的 LLMs 回答充当黄金标准答案的代理，以更准确地测量幻觉，从而减少存在的威胁并提高 LLMs 的可靠性和信誉。