There is increasing interest in the use of multimodal data in various web
applications, such as digital advertising and e-commerce. Typical methods for
extracting important information from multimodal data rely on a mid-fusion
architecture that combines the feature representations from multiple encoders.
However, as the number of modalities increases, several potential problems with
the mid-fusion model structure arise, such as an increase in the dimensionality
of the concatenated multimodal features and missing modalities. To address
these problems, we propose a new concept that considers multimodal inputs as a
set of sequences, namely, deep multimodal sequence sets (DM$^2$S$^2$). Our
set-aware concept consists of three components that capture the relationships
among multiple modalities: (a) a BERT-based encoder to handle the inter- and
intra-order of elements in the sequences, (b) intra-modality residual attention
(IntraMRA) to capture the importance of the elements in a modality, and (c)
inter-modality residual attention (InterMRA) to enhance the importance of
elements with modality-level granularity further. Our concept exhibits
performance that is comparable to or better than the previous set-aware models.
Furthermore, we demonstrate that the visualization of the learned InterMRA and
IntraMRA weights can provide an interpretation of the prediction results.

该研究提出了一种新的多模态输入概念 —— 深度多模态序列集，其包括三个组件来捕捉多个模态之间的关系，并使用 BERT-based 全局编码器和残差注意力来提取重要特征，该模型的结果可解释性强，并且具有与先前 set-aware 模型相媲美或更好的性能。