Multimodal learning has mainly focused on learning large models on, and
fusing feature representations from, different modalities for better
performances on downstream tasks. In this work, we take a detour from this
trend and study the intrinsic nature of multimodal data by asking the following
questions: 1) Can we learn more structured latent representations of general
multimodal data?; and 2) can we intuitively understand, both mathematically and
visually, what the latent representations capture? To answer 1), we propose a
general and lightweight framework, Multimodal Understanding Through Correlation
Maximization and Minimization (MUCMM), that can be incorporated into any large
pre-trained network. MUCMM learns both the common and individual
representations. The common representations capture what is common between the
modalities; the individual representations capture the unique aspect of the
modalities. To answer 2), we propose novel scores that summarize the learned
common and individual structures and visualize the score gradients with respect
to the input, visually discerning what the different representations capture.
We further provide mathematical intuitions of the computed gradients in a
linear setting, and demonstrate the effectiveness of our approach through a
variety of experiments.

使用 Multimodal Understanding Through Correlation Maximization and Minimization（MUCMM） 框架，该框架可以轻松地嵌入到任何预训练网络中，学习模态数据的常见和个体表示，根据提出的方法，通过可视化分析过程给出了 MUCMM 学习结构化潜在表示的解释。