Large Language Models (LLMs) are revolutionizing various domains, yet
verifying their answers remains a significant challenge, especially for
intricate open-ended tasks such as consolidation, summarization, and extraction
of knowledge. In this work, we propose CheckEmbed: an accurate, scalable, and
simple LLM verification approach. CheckEmbed is driven by a straightforward yet
powerful idea: in order to compare LLM solutions to one another or to the
ground-truth, compare their corresponding answer-level embeddings obtained with
a model such as GPT Text Embedding Large. This reduces a complex textual answer
to a single embedding, facilitating straightforward, fast, and meaningful
verification. We develop a comprehensive verification pipeline implementing the
CheckEmbed methodology. The CheckEmbed pipeline also comes with metrics for
assessing the truthfulness of the LLM answers, such as embedding heatmaps and
their summaries. We show how to use these metrics for deploying practical
engines that decide whether an LLM answer is satisfactory or not. We apply the
pipeline to real-world document analysis tasks, including term extraction and
document summarization, showcasing significant improvements in accuracy,
cost-effectiveness, and runtime performance compared to existing token-,
sentence-, and fact-level schemes such as BERTScore or SelfCheckGPT.

提出了一个简单而有力的证实方法，通过比较基于大型语言模型的答案级嵌入来验证模型的答案，以实现诸如汇总、摘要和知识提取等复杂开放式任务的准确性和可扩展性，并展示了在术语提取和文档摘要等实际任务中，与现有方案相比，精度、性价比和运行性能都有显著提高。