There is increasing interest in the use of multimodal data in various web
applications, such as digital advertising and e-commerce. Typical methods for
extracting important information from multimodal data rely on a mid-fusion
architecture that combines the feature representations from multiple encoders.
However, as the number of modalities increases, several potential problems with
the mid-fusion model structure arise, such as an increase in the dimensionality
of the concatenated multimodal features and missing modalities. To address
these problems, we propose a new concept that considers multimodal inputs as a
set of sequences, namely, deep multimodal sequence sets (DM$^2$S$^2$). Our
set-aware concept consists of three components that capture the relationships
among multiple modalities: (a) a BERT-based encoder to handle the inter- and
intra-order of elements in the sequences, (b) intra-modality residual attention
(IntraMRA) to capture the importance of the elements in a modality, and (c)
inter-modality residual attention (InterMRA) to enhance the importance of
elements with modality-level granularity further. Our concept exhibits
performance that is comparable to or better than the previous set-aware models.
Furthermore, we demonstrate that the visualization of the learned InterMRA and
IntraMRA weights can provide an interpretation of the prediction results.

该研究提出了一种新的多模态输入概念 —— 深度多模态序列集，其包括三个组件来捕捉多个模态之间的关系，并使用 BERT-based 全局编码器和残差注意力来提取重要特征，该模型的结果可解释性强，并且具有与先前 set-aware 模型相媲美或更好的性能。

DM$^2$S$^2$: 深度多模态序列集，具有分层模态关注

DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention

Social media plays a significant role in disaster management by providing
valuable data about affected people, donations and help requests. Recent
studies highlight the need to filter information on social media into
fine-grained content labels. However, identifying useful information from
massive amounts of social media posts during a crisis is a challenging task. In
this paper, we propose I-AID, a multimodel approach to automatically categorize
tweets into multi-label information types and filter critical information from
the enormous volume of social media data. I-AID incorporates three main
components: i) a BERT-based encoder to capture the semantics of a tweet and
represent as a low-dimensional vector, ii) a graph attention network (GAT) to
apprehend correlations between tweets' words/entities and the corresponding
information types, and iii) a Relation Network as a learnable distance metric
to compute the similarity between tweets and their corresponding information
types in a supervised way. We conducted several experiments on two real
publicly-available datasets. Our results indicate that I-AID outperforms
state-of-the-art approaches in terms of weighted average F1 score by +6% and
+4% on the TREC-IS dataset and COVID-19 Tweets, respectively.

本文提出 I-AID，这是一种多模型方法，可将推文自动分类为多标签信息类型，并从海量的社交媒体数据中过滤关键信息。I-AID 包括三个主要组件：基于 BERT 的编码器、图注意力网络（GAT）和可学习距离度量的关系网络。经过多次实验，结果表明在 TREC-IS 数据集和 COVID-19 推文上，I-AID 在加权平均 F1 得分方面的表现优于现有的最新方法 6％和 4％。