This paper delves into the formidable challenge of cross-domain
generalization in multimodal hate meme detection, presenting compelling
findings. We provide enough pieces of evidence supporting the hypothesis that
only the textual component of hateful memes enables the existing multimodal
classifier to generalize across different domains, while the image component
proves highly sensitive to a specific training dataset. The evidence includes
demonstrations showing that hate-text classifiers perform similarly to
hate-meme classifiers in a zero-shot setting. Simultaneously, the introduction
of captions generated from images of memes to the hate-meme classifier worsens
performance by an average F1 of 0.02. Through blackbox explanations, we
identify a substantial contribution of the text modality (average of 83%),
which diminishes with the introduction of meme's image captions (52%).
Additionally, our evaluation on a newly created confounder dataset reveals
higher performance on text confounders as compared to image confounders with an
average $\Delta$F1 of 0.18.

通过相关证据，该研究论文探讨了多模态仇恨迷因检测中跨领域泛化的巨大挑战，并发现只有具有仇恨性质的迷因的文本组成部分能够使现有的多模态分类器在不同领域中泛化，而图像组成部分对特定的训练数据集非常敏感。同时，研究通过黑盒解释，确定了文本模态的重要贡献程度为 83%，而引入迷因图像标题后该贡献程度减少至 52%。另外，对新创建的混淆数据集进行的评估显示，与图像混淆因素相比，文本混淆因素具有更高的性能，平均 ΔF1 为 0.18。