Learning multimodal representations is a fundamentally complex research
problem due to the presence of multiple heterogeneous sources of information.
Although the presence of multiple modalities provides additional valuable
information, there are two key challenges to address when learning from
multimodal data: 1) models must learn the complex intra-modal and cross-modal
interactions for prediction and 2) models must be robust to unexpected missing
or noisy modalities during testing. In this paper, we propose to optimize for a
joint generative-discriminative objective across multimodal data and labels. We
introduce a model that factorizes representations into two sets of independent
factors: multimodal discriminative and modality-specific generative factors.
Multimodal discriminative factors are shared across all modalities and contain
joint multimodal features required for discriminative tasks such as sentiment
prediction. Modality-specific generative factors are unique for each modality
and contain the information required for generating data. Experimental results
show that our model is able to learn meaningful multimodal representations that
achieve state-of-the-art or competitive performance on six multimodal datasets.
Our model demonstrates flexible generative capabilities by conditioning on
independent factors and can reconstruct missing modalities without
significantly impacting performance. Lastly, we interpret our factorized
representations to understand the interactions that influence multimodal
learning.

本文提出一种优化跨多模态数据和标签的联合生成 - 判别目标函数的模型，将表示分解为多模态判别和模态特异性生成两组因子，实验结果表明该模型能够学习到有意义的多模态表示，并取得了领先水平的性能。