Multi-modal fact verification has become an important but challenging issue
on social media due to the mismatch between the text and images in the
misinformation of news content, which has been addressed by considering
cross-modalities to identify the veracity of the news in recent years. In this
paper, we propose the Pre-CoFactv2 framework with new parameter-efficient
foundation models for modeling fine-grained text and input embeddings with
lightening parameters, multi-modal multi-type fusion for not only capturing
relations for the same and different modalities but also for different types
(i.e., claim and document), and feature representations for explicitly
providing metadata for each sample. In addition, we introduce a unified
ensemble method to boost model performance by adjusting the importance of each
trained model with not only the weights but also the powers. Extensive
experiments show that Pre-CoFactv2 outperforms Pre-CoFact by a large margin and
achieved new state-of-the-art results at the Factify challenge at AAAI 2023. We
further illustrate model variations to verify the relative contributions of
different components. Our team won the first prize (F1-score: 81.82%) and we
made our code publicly available at
this https URL.

本文介绍了 Pre-CoFactv2 框架，使用多模态、多类型融合方法以及特征表示来实现文本与图像的多模态事实验证，并使用统一的集成方法提高模型性能， 最终在 AAAI 2023 的 Factify challenge 中获得了最高成绩和新的最佳结果。