Accurate text summarization is one of the most common and important tasks
performed by Large Language Models, where the costs of human review for an
entire document may be high, but the costs of errors in summarization may be
even greater. We propose Detecting Errors through Ensembling Prompts (DEEP) -
an end-to-end large language model framework for detecting factual errors in
text summarization. Our framework uses a diverse set of LLM prompts to identify
factual inconsistencies, treating their outputs as binary features, which are
then fed into ensembling models. We then calibrate the ensembled models to
produce empirically accurate probabilities that a text is factually consistent
or free of hallucination. We demonstrate that prior models for detecting
factual errors in summaries perform significantly worse without optimizing the
thresholds on subsets of the evaluated dataset. Our framework achieves
state-of-the-art (SOTA) balanced accuracy on the AggreFact-XSUM FTSOTA,
TofuEval Summary-Level, and HaluEval Summarization benchmarks in detecting
factual errors within transformer-generated text summaries. It does so without
any fine-tuning of the language model or reliance on thresholding techniques
not available in practical settings.

用于检测大语言模型生成的文本摘要中的事实错误的一种框架，该框架通过使用多样的语言模型提示来识别事实不一致，并将其输出作为二进制特征进入集成模型，从而产生经验性准确的概率来判断文本是否在事实上一致或不含幻想。