Pervasive use of social media has become the emerging source for real-time
information (like images, text, or both) to identify various events. Despite
the rapid growth of image and text-based event classification, the
state-of-the-art (SOTA) models find it challenging to bridge the semantic gap
between features of image and text modalities due to inconsistent encoding.
Also, the black-box nature of models fails to explain the model's outcomes for
building trust in high-stakes situations such as disasters, pandemic.
Additionally, the word limit imposed on social media posts can potentially
introduce bias towards specific events. To address these issues, we proposed
CrisisKAN, a novel Knowledge-infused and Explainable Multimodal Attention
Network that entails images and texts in conjunction with external knowledge
from Wikipedia to classify crisis events. To enrich the context-specific
understanding of textual information, we integrated Wikipedia knowledge using
proposed wiki extraction algorithm. Along with this, a guided cross-attention
module is implemented to fill the semantic gap in integrating visual and
textual data. In order to ensure reliability, we employ a model-specific
approach called Gradient-weighted Class Activation Mapping (Grad-CAM) that
provides a robust explanation of the predictions of the proposed model. The
comprehensive experiments conducted on the CrisisMMD dataset yield in-depth
analysis across various crisis-specific tasks and settings. As a result,
CrisisKAN outperforms existing SOTA methodologies and provides a novel view in
the domain of explainable multimodal event classification.

提出了 CrisisKAN，一种新颖的知识注入和可解释的多模态注意力网络，结合了来自维基百科的图像和文本以分类危机事件。实验证明，CrisisKAN 在解释性多模态事件分类领域超过了现有的最先进方法。