Current automated fact-checking (AFC) approaches commonly evaluate evidence either implicitly via the predicted verdicts or by comparing retrieved evidence with a predefined closed knowledge source, such as Wikipedia. However, these methods suffer from limitations, resulting from their reliance on evaluation metrics developed for different purposes and constraints imposed by closed knowledge sources. Recent advances in natural language generation (NLG) evaluation offer new possibilities for evidence assessment. In this work, we introduce Ev2R, an evaluation framework for AFC that comprises three types of approaches for evidence evaluation: reference-based, proxy-reference, and reference-less. We evaluate their effectiveness through agreement with human ratings and adversarial tests, and demonstrate that prompt-based scorers, particularly those leveraging LLMs and reference evidence, outperform traditional evaluation approaches.

本研究解决了现有自动化事实检查方法在证据评估中的局限性，提出了Ev2R评价框架，该框架包含三种证据评估方法：基于参考、代理参考和无参考。研究结果表明，采用大型语言模型和参考证据的提示评分方法在与人工评分的一致性和对抗性测试中表现优于传统评估方法，具有显著的影响力。

Ev2R：自动化事实检查中证据检索的评估