Multimodal intent recognition aims to leverage diverse modalities such as
expressions, body movements and tone of speech to comprehend user's intent,
constituting a critical task for understanding human language and behavior in
real-world multimodal scenarios. Nevertheless, the majority of existing methods
ignore potential correlations among different modalities and own limitations in
effectively learning semantic features from nonverbal modalities. In this
paper, we introduce a token-level contrastive learning method with
modality-aware prompting (TCL-MAP) to address the above challenges. To
establish an optimal multimodal semantic environment for text modality, we
develop a modality-aware prompting module (MAP), which effectively aligns and
fuses features from text, video and audio modalities with similarity-based
modality alignment and cross-modality attention mechanism. Based on the
modality-aware prompt and ground truth labels, the proposed token-level
contrastive learning framework (TCL) constructs augmented samples and employs
NT-Xent loss on the label token. Specifically, TCL capitalizes on the optimal
textual semantic insights derived from intent labels to guide the learning
processes of other modalities in return. Extensive experiments show that our
method achieves remarkable improvements compared to state-of-the-art methods.
Additionally, ablation analyses demonstrate the superiority of the
modality-aware prompt over the handcrafted prompt, which holds substantial
significance for multimodal prompt learning. The codes are released at
this https URL

引入了一个名为 TCL-MAP 的方法，通过引导其它模态学习过程，利用文本语义洞察力来提高多模态意图识别的效果。该方法在实验证明了其在准确度上相较于现有方法有显著提升，而模态感知提示对多模态提示学习具有重要意义。

多模态意图识别的令牌级对比学习与模态感知提示

Token-Level Contrastive Learning with Modality-Aware Prompting for  Multimodal Intent Recognition

Multimodal sentiment analysis is an important area for understanding the
user's internal states. Deep learning methods were effective, but the problem
of poor interpretability has gradually gained attention. Previous works have
attempted to use attention weights or vector distributions to provide
interpretability. However, their explanations were not intuitive and can be
influenced by different trained models. This study proposed a novel approach to
provide interpretability by converting nonverbal modalities into text
descriptions and by using large-scale language models for sentiment
predictions. This provides an intuitive approach to directly interpret what
models depend on with respect to making decisions from input texts, thus
significantly improving interpretability. Specifically, we convert descriptions
based on two feature patterns for the audio modality and discrete action units
for the facial modality. Experimental results on two sentiment analysis tasks
demonstrated that the proposed approach maintained, or even improved
effectiveness for sentiment analysis compared to baselines using conventional
features, with the highest improvement of 2.49% on the F1 score. The results
also showed that multimodal descriptions have similar characteristics on fusing
modalities as those of conventional fusion methods. The results demonstrated
that the proposed approach is interpretable and effective for multimodal
sentiment analysis.

本研究提出了一种新方法，将非语言模态转化为文本描述，并使用大规模语言模型进行情感预测，从而提高情感分析的可解释性。在两个情感分析任务上的实验结果表明，该方法在维持或甚至提高了情感分析的效率的同时，使解释更为直接。