We present the Multi-Modal Discussion Transformer (mDT), a novel multi-modal
graph-based transformer model for detecting hate speech in online social
networks. In contrast to traditional text-only methods, our approach to
labelling a comment as hate speech centers around the holistic analysis of text
and images. This is done by leveraging graph transformers to capture the
contextual relationships in the entire discussion that surrounds a comment,
with interwoven fusion layers to combine text and image embeddings instead of
processing different modalities separately. We compare the performance of our
model to baselines that only process text; we also conduct extensive ablation
studies. We conclude with future work for multimodal solutions to deliver
social value in online contexts, arguing that capturing a holistic view of a
conversation greatly advances the effort to detect anti-social behavior.

我们提出了一种新颖的多模态基于图的 Transformer 模型（mDT），用于检测在线社交网络中的仇恨言论。与传统的仅处理文本的方法相比，我们的方法围绕整体分析文本和图像对评论进行仇恨言论标记。为了组合文本和图像嵌入而不是单独处理不同的模态，我们利用图 Transformer 来捕捉整个讨论中的上下文关系，并插入融合层。我们将模型的性能与仅处理文本的基准进行比较，并进行了广泛的消融研究。最后，我们提出了未来的工作，探讨多模态解决方案在在线环境中提供社会价值，并认为捕捉对话的整体观点大大推进了检测反社会行为的努力。