One of the challenges of developing a summarization model arises from the
difficulty in measuring the factual inconsistency of the generated text. In
this study, we reinterpret the decoder overconfidence-regularizing objective
suggested in (Miao et al., 2021) as a hallucination risk measurement to better
estimate the quality of generated summaries. We propose a reference-free
metric, HaRiM+, which only requires an off-the-shelf summarization model to
compute the hallucination risk based on token likelihoods. Deploying it
requires no additional training of models or ad-hoc modules, which usually need
alignment to human judgments. For summary-quality estimation, HaRiM+ records
state-of-the-art correlation to human judgment on three summary-quality
annotation sets: FRANK, QAGS, and SummEval. We hope that our work, which merits
the use of summarization models, facilitates the progress of both automated
evaluation and generation of summary.

本研究提出一个基于口令似然性的无参考度量，名为 HaRiM+，用于衡量生成摘要的虚假程度，可以用于衡量摘要质量，且在三个摘要质量注释数据集中均表现出最先进的与人工判断相关性，希望这项工作有助于推动自动摘要模型的发展。