Hateful memes have become a significant concern on the Internet, necessitating robust automated detection systems. While large multimodal models have shown strong generalization across various tasks, they exhibit poor generalization to hateful meme detection due to the dynamic nature of memes tied to emerging social trends and breaking news. Recent work further highlights the limitations of conventional supervised fine-tuning for large multimodal models in this context. To address these challenges, we propose Large Multimodal Model Retrieval-Guided Contrastive Learning (LMM-RGCL), a novel two-stage fine-tuning framework designed to improve both in-domain accuracy and cross-domain generalization. Experimental results on six widely used meme classification datasets demonstrate that LMM-RGCL achieves state-of-the-art performance, outperforming agent-based systems such as VPD-PALI-X-55B. Furthermore, our method effectively generalizes to out-of-domain memes under low-resource settings, surpassing models like GPT-4o.

本研究针对当前大型多模态模型在仇恨谣言检测中的泛化能力不足这一问题，提出了一种新颖的两阶段微调框架——大型多模态模型检索引导对比学习（LMM-RGCL）。实验结果表明，LMM-RGCL在六个常用的谣言分类数据集上表现出色，不仅提高了领域内的准确性，也在低资源环境下有效地对跨领域的谣言进行了泛化，超越了现有模型（如GPT-4o）的表现。

针对仇恨谣言检测的大型多模态模型改进微调