This paper describes our participant system for the multi-modal fact
verification (Factify) challenge at AAAI 2022. Despite the recent advance in
text based verification techniques and large pre-trained multimodal models
cross vision and language, very limited work has been done in applying
multimodal techniques to automate fact checking process, particularly
considering the increasing prevalence of claims and fake news about images and
videos on social media. In our work, the challenge is treated as multimodal
entailment task and framed as multi-class classification. Two baseline
approaches are proposed and explored including an ensemble model (combining two
uni-modal models) and a multi-modal attention network (modeling the interaction
between image and text pair from claim and evidence document). We conduct
several experiments investigating and benchmarking different SoTA pre-trained
transformers and vision models in this work. Our best model is ranked first in
leaderboard which obtains a weighted average F-measure of 0.77 on both
validation and test set. Exploratory analysis of dataset is also carried out on
the Factify data set and uncovers salient patterns and issues (e.g., word
overlapping, visual entailment correlation, source bias) that motivates our
hypothesis. Finally, we highlight challenges of the task and multimodal dataset
for future research.

本文描述了我们在 2022 AAAI 多模态事实验证（Factify）挑战上的参与系统。 我们处理该挑战为多模态蕴含任务，作为多类分类，并提出并探索了两种基线方法，包括集成模型和多模态注意力网络。 我们在此工作中测试了不同 SoTA 预训练变换器和视觉模型。 最佳模型在排行榜中排名第一， 平均 F - 度量值为 0.77。 最后，我们强调了未来研究的任务和多模态数据集的挑战。