Recently, large language models (LLMs) have taken the spotlight in natural
language processing. Further, integrating LLMs with vision enables the users to
explore more emergent abilities in multimodality. Visual language models
(VLMs), such as LLaVA, Flamingo, or GPT-4, have demonstrated impressive
performance on various visio-linguistic tasks. Consequently, there are enormous
applications of large models that could be potentially used on social media
platforms. Despite that, there is a lack of related work on detecting or
correcting hateful memes with VLMs. In this work, we study the ability of VLMs
on hateful meme detection and hateful meme correction tasks with zero-shot
prompting. From our empirical experiments, we show the effectiveness of the
pretrained LLaVA model and discuss its strengths and weaknesses in these tasks.

本研究探讨了使用视觉语言模型进行仇恨表情包检测和修正任务的能力，并通过实证实验证明了预训练的 LLaVA 模型在这些任务中的有效性和讨论了其优缺点。

大规模视觉语言模型中的多模态违法言论检测与纠正

Detecting and Correcting Hate Speech in Multimodal Memes with Large  Visual Language Model

Hateful meme detection is a challenging multimodal task that requires
comprehension of both vision and language, as well as cross-modal interactions.
Recent studies have tried to fine-tune pre-trained vision-language models
(PVLMs) for this task. However, with increasing model sizes, it becomes
important to leverage powerful PVLMs more efficiently, rather than simply
fine-tuning them. Recently, researchers have attempted to convert meme images
into textual captions and prompt language models for predictions. This approach
has shown good performance but suffers from non-informative image captions.
Considering the two factors mentioned above, we propose a probing-based
captioning approach to leverage PVLMs in a zero-shot visual question answering
(VQA) manner. Specifically, we prompt a frozen PVLM by asking hateful
content-related questions and use the answers as image captions (which we call
Pro-Cap), so that the captions contain information critical for hateful content
detection. The good performance of models with Pro-Cap on three benchmarks
validates the effectiveness and generalization of the proposed method.

使用 Probing-Based Captioning 方法，通过向一个冻结的 PVLM 提问来生成图像标题，以解决令人讨厌的模因检测任务中的非信息性图像标题的问题，该方法在三个基准测试上验证了其有效性和泛化性。

Pro-Cap: 利用冻结的视觉语言模型进行令人讨厌的恶搞表情包检测

Pro-Cap: Leveraging a Frozen Vision-Language Model for Hateful Meme  Detection

Hateful meme detection is a new research area recently brought out that
requires both visual, linguistic understanding of the meme and some background
knowledge to performing well on the task. This technical report summarises the
first place solution of the Hateful Meme Detection Challenge 2020, which
extending state-of-the-art visual-linguistic transformers to tackle this
problem. At the end of the report, we also point out the shortcomings and
possible directions for improving the current methodology.

本报告总结了 2020 年针对仇恨表情包检测挑战赛首名解决方案，利用先进的视觉语言转换器来解决这个问题，并指出了目前方法的不足和改进方向。