Misinformation is now a major problem due to its potential high risks to our
core democratic and societal values and orders. Out-of-context misinformation
is one of the easiest and effective ways used by adversaries to spread viral
false stories. In this threat, a real image is re-purposed to support other
narratives by misrepresenting its context and/or elements. The internet is
being used as the go-to way to verify information using different sources and
modalities. Our goal is an inspectable method that automates this
time-consuming and reasoning-intensive process by fact-checking the
image-caption pairing using Web evidence. To integrate evidence and cues from
both modalities, we introduce the concept of 'multi-modal cycle-consistency
check'; starting from the image/caption, we gather textual/visual evidence,
which will be compared against the other paired caption/image, respectively.
Moreover, we propose a novel architecture, Consistency-Checking Network (CCN),
that mimics the layered human reasoning across the same and different
modalities: the caption vs. textual evidence, the image vs. visual evidence,
and the image vs. caption. Our work offers the first step and benchmark for
open-domain, content-based, multi-modal fact-checking, and significantly
outperforms previous baselines that did not leverage external evidence.

该研究提出了一种基于多模态证据的检查方法，通过对图像和标题进行事实检查，有效应对了当前社交网络中易传播的虚假信息。同时，引入了一种名为 'Consistency-Checking Network' 的新型架构，模拟了人类跨模态推理的过程。