Multimodal Language Analysis is a demanding area of research, since it is
associated with two requirements: combining different modalities and capturing
temporal information. During the last years, several works have been proposed
in the area, mostly centered around supervised learning in downstream tasks. In
this paper we propose extracting unsupervised Multimodal Language
representations that are universal and can be applied to different tasks.
Towards this end, we map the word-level aligned multimodal sequences to 2-D
matrices and then use Convolutional Autoencoders to learn embeddings by
combining multiple datasets. Extensive experimentation on Sentiment Analysis
(MOSEI) and Emotion Recognition (IEMOCAP) indicate that the learned
representations can achieve near-state-of-the-art performance with just the use
of a Logistic Regression algorithm for downstream classification. It is also
shown that our method is extremely lightweight and can be easily generalized to
other tasks and unseen data with small performance drop and almost the same
number of parameters. The proposed multimodal representation models are
open-sourced and will help grow the applicability of Multimodal Language.

提出了一种无监督多模态语言表示提取方法，利用卷积自编码器将单词级对齐的多模态序列映射到 2-D 矩阵中，证明其在情感分析和情绪识别领域可以达到接近最先进性能的结果。

使用卷积自编码器实现无监督多模态语言表征

Unsupervised Multimodal Language Representations using Convolutional  Autoencoders

Multimodal language analysis often considers relationships between features
based on text and those based on acoustical and visual properties. Text
features typically outperform non-text features in sentiment analysis or
emotion recognition tasks in part because the text features are derived from
advanced language models or word embeddings trained on massive data sources
while audio and video features are human-engineered and comparatively
underdeveloped. Given that the text, audio, and video are describing the same
utterance in different ways, we hypothesize that the multimodal sentiment
analysis and emotion recognition can be improved by learning (hidden)
correlations between features extracted from the outer product of text and
audio (we call this text-based audio) and analogous text-based video. This
paper proposes a novel model, the Interaction Canonical Correlation Network
(ICCN), to learn such multimodal embeddings. ICCN learns correlations between
all three modes via deep canonical correlation analysis (DCCA) and the proposed
embeddings are then tested on several benchmark datasets and against other
state-of-the-art multimodal embedding algorithms. Empirical results and
ablation studies confirm the effectiveness of ICCN in capturing useful
information from all three views.

本研究提出了一个名为交互规范相关网络（ICCN）的模型，利用文本、音频和视频的特征之间的相关性来提高多模态情感分析和情感识别的准确性，该模型通过深度规范相关分析（DCCA）来学习所有三种模式之间的关联，并在多个基准数据集中进行了测试。