Multi-modal aspect-based sentiment classification (MABSC) is an emerging
classification task that aims to classify the sentiment of a given target such
as a mentioned entity in data with different modalities. In typical multi-modal
data with text and image, previous approaches do not make full use of the
fine-grained semantics of the image, especially in conjunction with the
semantics of the text and do not fully consider modeling the relationship
between fine-grained image information and target, which leads to insufficient
use of image and inadequate to identify fine-grained aspects and opinions. To
tackle these limitations, we propose a new framework SeqCSG including a method
to construct sequential cross-modal semantic graphs and an encoder-decoder
model. Specifically, we extract fine-grained information from the original
image, image caption, and scene graph, and regard them as elements of the
cross-modal semantic graph as well as tokens from texts. The cross-modal
semantic graph is represented as a sequence with a multi-modal visible matrix
indicating relationships between elements. In order to effectively utilize the
cross-modal semantic graph, we propose an encoder-decoder method with a target
prompt template. Experimental results show that our approach outperforms
existing methods and achieves the state-of-the-art on two standard datasets
MABSC. Further analysis demonstrates the effectiveness of each component and
our model can implicitly learn the correlation between the target and
fine-grained information of the image.

本文提出了一种新的框架 SeqCSG，其中包括构建顺序跨模态语义图和编码器 - 解码器模型方法，用于有效利用跨模态语义图进行多模态方面的情感分类，实验结果表明，我们的方法优于现有方法并在两个标准数据集中实现了最先进水平。