Multimodal sentiment analysis is an important area for understanding the
user's internal states. Deep learning methods were effective, but the problem
of poor interpretability has gradually gained attention. Previous works have
attempted to use attention weights or vector distributions to provide
interpretability. However, their explanations were not intuitive and can be
influenced by different trained models. This study proposed a novel approach to
provide interpretability by converting nonverbal modalities into text
descriptions and by using large-scale language models for sentiment
predictions. This provides an intuitive approach to directly interpret what
models depend on with respect to making decisions from input texts, thus
significantly improving interpretability. Specifically, we convert descriptions
based on two feature patterns for the audio modality and discrete action units
for the facial modality. Experimental results on two sentiment analysis tasks
demonstrated that the proposed approach maintained, or even improved
effectiveness for sentiment analysis compared to baselines using conventional
features, with the highest improvement of 2.49% on the F1 score. The results
also showed that multimodal descriptions have similar characteristics on fusing
modalities as those of conventional fusion methods. The results demonstrated
that the proposed approach is interpretable and effective for multimodal
sentiment analysis.

本研究提出了一种新方法，将非语言模态转化为文本描述，并使用大规模语言模型进行情感预测，从而提高情感分析的可解释性。在两个情感分析任务上的实验结果表明，该方法在维持或甚至提高了情感分析的效率的同时，使解释更为直接。

基于大规模语言模型的可解释的多模态情感分析

Interpretable multimodal sentiment analysis based on textual modality  descriptions by using large-scale language models

Aspect based sentiment analysis (ABSA) aims to identify the sentiment
polarity towards the given aspect in a sentence, while previous models
typically exploit an aspect-independent (weakly associative) encoder for
sentence representation generation. In this paper, we propose a novel
Aspect-Guided Deep Transition model, named AGDT, which utilizes the given
aspect to guide the sentence encoding from scratch with the specially-designed
deep transition architecture. Furthermore, an aspect-oriented objective is
designed to enforce AGDT to reconstruct the given aspect with the generated
sentence representation. In doing so, our AGDT can accurately generate
aspect-specific sentence representation, and thus conduct more accurate
sentiment predictions. Experimental results on multiple SemEval datasets
demonstrate the effectiveness of our proposed approach, which significantly
outperforms the best reported results with the same setting.

本文提出了一种新颖的 Aspect-Guided Deep Transition 模型（AGDT），该模型通过专门设计的深层转换结构从头开始利用所给定的方面来指导句子编码，以重构给定的方面与生成的句子表示相结合的面向方面的目标，从而能够准确生成面向方面的句子表示形式，进而进行更准确的情感预测，并通过多个 SemEval 数据集的实验结果表明，我们提出的方法明显优于最佳报道结果。