Multimodal hateful content detection is a challenging task that requires
complex reasoning across visual and textual modalities. Therefore, creating a
meaningful multimodal representation that effectively captures the interplay
between visual and textual features through intermediate fusion is critical.
Conventional fusion techniques are unable to attend to the modality-specific
features effectively. Moreover, most studies exclusively concentrated on
English and overlooked other low-resource languages. This paper proposes a
context-aware attention framework for multimodal hateful content detection and
assesses it for both English and non-English languages. The proposed approach
incorporates an attention layer to meaningfully align the visual and textual
features. This alignment enables selective focus on modality-specific features
before fusing them. We evaluate the proposed approach on two benchmark hateful
meme datasets, viz. MUTE (Bengali code-mixed) and MultiOFF (English).
Evaluation results demonstrate our proposed approach's effectiveness with
F1-scores of $69.7$% and $70.3$% for the MUTE and MultiOFF datasets. The scores
show approximately $2.5$% and $3.2$% performance improvement over the
state-of-the-art systems on these datasets. Our implementation is available at
this https URL

提出了一种上下文感知的注意力框架用于多模态仇恨内容检测，对英语和非英语语言进行了评估，并通过对 MUTE 和 MultiOFF 数据集的 F1 分数进行评估，证明了该方法的有效性。

对齐先于关注：为多模态仇恨内容检测对齐视觉和文字特征

Align before Attend: Aligning Visual and Textual Features for Multimodal  Hateful Content Detection

We consider the problem of estimating a consensus community structure by
combining information from multiple layers of a multi-layer network using
methods based on the spectral clustering or a low-rank matrix factorization. As
a general theme, these "intermediate fusion" methods involve obtaining a low
column rank matrix by optimizing an objective function and then using the
columns of the matrix for clustering. However, the theoretical properties of
these methods remain largely unexplored. In the absence of statistical
guarantees on the objective functions, it is difficult to determine if the
algorithms optimizing the objectives will return good community structures. We
investigate the consistency properties of the global optimizer of some of these
objective functions under the multi-layer stochastic blockmodel. For this
purpose, we derive several new asymptotic results showing consistency of the
intermediate fusion techniques along with the spectral clustering of mean
adjacency matrix under a high dimensional setup, where the number of nodes, the
number of layers and the number of communities of the multi-layer graph grow.
Our numerical study shows that the intermediate fusion techniques outperform
late fusion methods, namely spectral clustering on aggregate spectral kernel
and module allegiance matrix in sparse networks, while they outperform the
spectral clustering of mean adjacency matrix in multi-layer networks that
contain layers with both homophilic and heterophilic communities.

本文探讨了利用基于谱聚类或低秩矩阵分解的中间融合方法，结合多层网络的信息来估计共识社区结构的问题。在高维设置下，它们可以有效地一致且精确地估计社区结构。