Hateful meme detection is a challenging multimodal task that requires
comprehension of both vision and language, as well as cross-modal interactions.
Recent studies have tried to fine-tune pre-trained vision-language models
(PVLMs) for this task. However, with increasing model sizes, it becomes
important to leverage powerful PVLMs more efficiently, rather than simply
fine-tuning them. Recently, researchers have attempted to convert meme images
into textual captions and prompt language models for predictions. This approach
has shown good performance but suffers from non-informative image captions.
Considering the two factors mentioned above, we propose a probing-based
captioning approach to leverage PVLMs in a zero-shot visual question answering
(VQA) manner. Specifically, we prompt a frozen PVLM by asking hateful
content-related questions and use the answers as image captions (which we call
Pro-Cap), so that the captions contain information critical for hateful content
detection. The good performance of models with Pro-Cap on three benchmarks
validates the effectiveness and generalization of the proposed method.

使用 Probing-Based Captioning 方法，通过向一个冻结的 PVLM 提问来生成图像标题，以解决令人讨厌的模因检测任务中的非信息性图像标题的问题，该方法在三个基准测试上验证了其有效性和泛化性。