As computer-generated content and deepfakes make steady improvements,
semantic approaches to multimedia forensics will become more important. In this
paper, we introduce a novel classification architecture for identifying
semantic inconsistencies between video appearance and text caption in social
media news posts. We develop a multi-modal fusion framework to identify
mismatches between videos and captions in social media posts by leveraging an
ensemble method based on textual analysis of the caption, automatic audio
transcription, semantic video analysis, object detection, named entity
consistency, and facial verification. To train and test our approach, we curate
a new video-based dataset of 4,000 real-world Facebook news posts for analysis.
Our multi-modal approach achieves 60.5% classification accuracy on random
mismatches between caption and appearance, compared to accuracy below 50% for
uni-modal models. Further ablation studies confirm the necessity of fusion
across modalities for correctly identifying semantic inconsistencies.

本研究提出了一种用于识别社交媒体新闻帖子中视频外观和文本标题之间语义不一致的分类架构，通过使用基于文本分析、自动音频转录、语义视频分析、对象检测、命名实体一致性和面部验证的多模态融合框架来发现社交媒体帖子中的视频和标题之间的不匹配。通过训练和测试基于 4,000 个 Facebook 新闻帖子的新视频数据集，我们的多模态方法在随机标题和外观不匹配的情况下实现了 60.5％的分类准确性，而单模分别准确率不到 50％。进一步的消融研究证实了跨模态融合对于正确识别语义不一致性的必要性。