This work proposes a new challenge set for multimodal classification,
focusing on detecting hate speech in multimodal memes. It is constructed such
that unimodal models struggle and only multimodal models can succeed: difficult
examples ("benign confounders") are added to the dataset to make it hard to
rely on unimodal signals. The task requires subtle reasoning, yet is
straightforward to evaluate as a binary classification problem. We provide
baseline performance numbers for unimodal models, as well as for multimodal
models with various degrees of sophistication. We find that state-of-the-art
methods perform poorly compared to humans (64.73% vs. 84.7% accuracy),
illustrating the difficulty of the task and highlighting the challenge that
this important problem poses to the community.

该研究提出了一个新的挑战，针对多模态模因素进行分类，专注于检测多模态模因素中的仇恨言论，并构造了一个难以依靠单模态信号的数据集，要求进行微妙的推理，证明了该任务的难度，并强调这个重要问题对社会的挑战。