Decoding methods for large language models (LLMs) usually struggle with the
tradeoff between ensuring factuality and maintaining diversity. For example, a
higher p threshold in the nucleus (top-p) sampling increases the diversity but
decreases the factuality, and vice versa. In this paper, we propose REAL
(Residual Entropy from Asymptotic Line) sampling, a decoding method that
achieves improved factuality and diversity over nucleus sampling by predicting
an adaptive threshold of $p$. Specifically, REAL sampling predicts the
step-wise likelihood of an LLM to hallucinate, and lowers the p threshold when
an LLM is likely to hallucinate. Otherwise, REAL sampling increases the p
threshold to boost the diversity. To predict the step-wise hallucination
likelihood without supervision, we construct a Token-level Hallucination
Forecasting (THF) model to predict the asymptotic entropy (i.e., inherent
uncertainty) of the next token by extrapolating the next-token entropies from a
series of LLMs with different sizes. If a LLM's entropy is higher than the
asymptotic entropy (i.e., the LLM is more uncertain than it should be), the THF
model predicts a high hallucination hazard, which leads to a lower p threshold
in REAL sampling. In the FactualityPrompts benchmark, we demonstrate that REAL
sampling based on a 70M THF model can substantially improve the factuality and
diversity of 7B LLMs simultaneously, judged by both retrieval-based metrics and
human evaluation. After combined with contrastive decoding, REAL sampling
outperforms 9 sampling methods, and generates texts that are more factual than
the greedy sampling and more diverse than the nucleus sampling with $p=0.5$.
Furthermore, the predicted asymptotic entropy is also a useful unsupervised
signal for hallucination detection tasks.

本文提出了一种名为 REAL（Residual Entropy from Asymptotic Line）采样的解码方法，通过预测自适应阈值 $p$，实现了比核心采样更好的准确性和多样性。实验结果表明，结合对比解码后，REAL 采样优于 9 种采样方法，生成的文本既比贪婪采样更准确，又比 $p=0.5$ 的核心采样更多样。此外，预测的渐近熵也是幻觉检测任务中一种有用的无监督信号。