Designing an effective representation learning method for multimodal sentiment analysis tasks is a crucial research direction. The challenge lies in learning both shared and private information in a complete modal representation, which is difficult with uniform multimodal labels and a raw feature fusion approach. In this work, we propose a deep modal shared information learning module based on the covariance matrix to capture the shared information between modalities. Additionally, we use a label generation module based on a self-supervised learning strategy to capture the private information of the modalities. Our module is plug-and-play in multimodal tasks, and by changing the parameterization, it can adjust the information exchange relationship between the modes and learn the private or shared information between the specified modes. We also employ a multi-task learning strategy to help the model focus its attention on the modal differentiation training data. We provide a detailed formulation derivation and feasibility proof for the design of the deep modal shared information learning module. We conduct extensive experiments on three common multimodal sentiment analysis baseline datasets, and the experimental results validate the reliability of our model. Furthermore, we explore more combinatorial techniques for the use of the module. Our approach outperforms current state-of-the-art methods on most of the metrics of the three public datasets.

本研究提出了一种基于深度学习的多模态情感分析的表示学习方法，使用协方差矩阵来捕捉模态之间的共享信息，同时采用基于自监督学习策略的标签生成模块来捕捉模态的私有信息，并使用多任务学习来帮助模型关注模态差异训练数据。实验结果表明，该方法在多个公共数据集上表现优于现有的同类方法。

多模态情感分析中深度模态对齐和自监督多任务学习的共享和私有信息学习