In the battle against widespread online misinformation, a growing problem is
text-image inconsistency, where images are misleadingly paired with texts with
different intent or meaning. Existing classification-based methods for
text-image inconsistency can identify contextual inconsistencies but fail to
provide explainable justifications for their decisions that humans can
understand. Although more nuanced, human evaluation is impractical at scale and
susceptible to errors. To address these limitations, this study introduces
D-TIIL (Diffusion-based Text-Image Inconsistency Localization), which employs
text-to-image diffusion models to localize semantic inconsistencies in text and
image pairs. These models, trained on large-scale datasets act as ``omniscient"
agents that filter out irrelevant information and incorporate background
knowledge to identify inconsistencies. In addition, D-TIIL uses text embeddings
and modified image regions to visualize these inconsistencies. To evaluate
D-TIIL's efficacy, we introduce a new TIIL dataset containing 14K consistent
and inconsistent text-image pairs. Unlike existing datasets, TIIL enables
assessment at the level of individual words and image regions and is carefully
designed to represent various inconsistencies. D-TIIL offers a scalable and
evidence-based approach to identifying and localizing text-image inconsistency,
providing a robust framework for future research combating misinformation.

为了解决在线误导信息的问题，该研究引入了 D-TIIL（基于扩散的文本 - 图像不一致性定位），采用文本对图像扩散模型来定位文本和图像对中的语义不一致性，并通过文本嵌入和修改后的图像区域进行可视化。