The increasing rate at which scientific knowledge is discovered and health
claims shared online has highlighted the importance of developing efficient
fact-checking systems for scientific claims. The usual setting for this task in
the literature assumes that the documents containing the evidence for claims
are already provided and annotated or contained in a limited corpus. This
renders the systems unrealistic for real-world settings where knowledge sources
with potentially millions of documents need to be queried to find relevant
evidence. In this paper, we perform an array of experiments to test the
performance of open-domain claim verification systems. We test the final
verdict prediction of systems on four datasets of biomedical and health claims
in different settings. While keeping the pipeline's evidence selection and
verdict prediction parts constant, document retrieval is performed over three
common knowledge sources (PubMed, Wikipedia, Google) and using two different
information retrieval techniques. We show that PubMed works better with
specialized biomedical claims, while Wikipedia is more suited for everyday
health concerns. Likewise, BM25 excels in retrieval precision, while semantic
search in recall of relevant evidence. We discuss the results, outline frequent
retrieval patterns and challenges, and provide promising future directions.

本研究通过一系列实验测试了开放领域的声明验证系统的性能，在不同设置下使用三个常见知识来源（PubMed、Wikipedia、Google）和两种不同的信息检索技术，对四个生物医学和健康声明的最终预测进行了测试。结果显示，对于专门的生物医学声明，PubMed 表现更好，而对于日常健康问题，Wikipedia 更适合；而在检索精度方面，BM25 表现出色，而在相关证据的回收方面，语义搜索更具优势。在讨论结果、概述频繁的检索模式和挑战，并提供有前途的未来方向。

开放领域科学主张验证的知识源对比

Comparing Knowledge Sources for Open-Domain Scientific Claim  Verification

Automatic fact verification has received significant attention recently.
Contemporary automatic fact-checking systems focus on estimating truthfulness
using numerical scores which are not human-interpretable. A human fact-checker
generally follows several logical steps to verify a verisimilitude claim and
conclude whether it is truthful or a mere masquerade. Popular fact-checking
websites follow a common structure for fact categorization such as half true,
half false, false, pants on fire, etc. Therefore, it is necessary to have an
aspect-based (which part is true and which part is false) explainable system
that can assist human fact-checkers in asking relevant questions related to a
fact, which can then be validated separately to reach a final verdict. In this
paper, we propose a 5W framework (who, what, when, where, and why) for
question-answer-based fact explainability. To that end, we have gathered a
semi-automatically generated dataset called FACTIFY-5WQA, which consists of
395, 019 facts along with relevant 5W QAs underscoring our major contribution
to this paper. A semantic role labeling system has been utilized to locate 5Ws,
which generates QA pairs for claims using a masked language model. Finally, we
report a baseline QA system to automatically locate those answers from evidence
documents, which can be served as the baseline for future research in this
field. Lastly, we propose a robust fact verification system that takes
paraphrased claims and automatically validates them. The dataset and the
baseline model are available at this https URL

本文提出了一个基于 5W 问答的可解释性事实验证框架，使用语义角色标记系统定位 5W 并生成 QA 对，我们获得了一个半自动生成的 FACTIFY-5WQA 数据集，并提出了一个基线 QA 系统自动定位从证据文件中的答案。最后，我们提出了一个强大的事实验证系统，可以对改写的声明进行自动验证。