Automatic detection of multimodal misinformation has gained a widespread
attention recently. However, the potential of powerful Large Language Models
(LLMs) for multimodal misinformation detection remains underexplored. Besides,
how to teach LLMs to interpret multimodal misinformation in cost-effective and
accessible way is still an open question. To address that, we propose MMIDR, a
framework designed to teach LLMs in providing fluent and high-quality textual
explanations for their decision-making process of multimodal misinformation. To
convert multimodal misinformation into an appropriate instruction-following
format, we present a data augmentation perspective and pipeline. This pipeline
consists of a visual information processing module and an evidence retrieval
module. Subsequently, we prompt the proprietary LLMs with processed contents to
extract rationales for interpreting the authenticity of multimodal
misinformation. Furthermore, we design an efficient knowledge distillation
approach to distill the capability of proprietary LLMs in explaining multimodal
misinformation into open-source LLMs. To explore several research questions
regarding the performance of LLMs in multimodal misinformation detection tasks,
we construct an instruction-following multimodal misinformation dataset and
conduct comprehensive experiments. The experimental findings reveal that our
MMIDR exhibits sufficient detection performance and possesses the capacity to
provide compelling rationales to support its assessments.

提出了 MMIDR 框架，用于教授大型语言模型在多模态错误信息检测中提供流畅、高质量的文本解释，通过数据增强和流程设计，将多模态错误信息转化为适当的指令遵循格式，再利用知识蒸馏方法将专有模型的解释能力传递给开源模型，实验证明 MMIDR 具有足够的检测性能，并能够提供有力的解释支持。