Misinformation emerges in times of uncertainty when credible information is
limited. This is challenging for NLP-based fact-checking as it relies on
counter-evidence, which may not yet be available. Despite increasing interest
in automatic fact-checking, it is still unclear if automated approaches can
realistically refute harmful real-world misinformation. Here, we contrast and
compare NLP fact-checking with how professional fact-checkers combat
misinformation in the absence of counter-evidence. In our analysis, we show
that, by design, existing NLP task definitions for fact-checking cannot refute
misinformation as professional fact-checkers do for the majority of claims. We
then define two requirements that the evidence in datasets must fulfill for
realistic fact-checking: It must be (1) sufficient to refute the claim and (2)
not leaked from existing fact-checking articles. We survey existing
fact-checking datasets and find that all of them fail to satisfy both criteria.
Finally, we perform experiments to demonstrate that models trained on a
large-scale fact-checking dataset rely on leaked evidence, which makes them
unsuitable in real-world scenarios. Taken together, we show that current NLP
fact-checking cannot realistically combat real-world misinformation because it
depends on unrealistic assumptions about counter-evidence in the data.

研究发现现有的 NLP 辟谣任务定义无法像专业辟谣人员那样驳斥绝大多数谣言，且现有数据集不满足辟谣的要求。同时，研究表明现有的大规模事实核查数据集的模型依赖于泄露的信息，因此无法在真实世界的情况下使用。因此，当前的 NLP 辟谣无法在真实世界的情况下有效应对虚假信息。