Human language is a rich multimodal signal consisting of spoken words, facial expressions, body gestures, and vocal intonations. Learning representations for these spoken utterances is a complex research problem due to the presence of multiple heterogeneous sources of information. Recent advances in multimodal learning have followed the general trend of building more complex models that utilize various attention, memory and recurrent components. In this paper, we propose two simple but strong baselines to learn embeddings of multimodal utterances. The first baseline assumes a conditional factorization of the utterance into unimodal factors. Each unimodal factor is modeled using the simple form of a likelihood function obtained via a linear transformation of the embedding. We show that the optimal embedding can be derived in closed form by taking a weighted average of the unimodal features. In order to capture richer representations, our second baseline extends the first by factorizing into unimodal, bimodal, and trimodal factors, while retaining simplicity and efficiency during learning and inference. From a set of experiments across two tasks, we show strong performance on both supervised and semi-supervised multimodal prediction, as well as significant (10 times) speedups over neural models during inference. Overall, we believe that our strong baseline models offer new benchmarking options for future research in multimodal learning.

本文提出了两种简单但强大的基线方法来学习多模式言论的嵌入表示。第一种基线方法假设言论被条件因式分解成单模态因子，并表明最优嵌入可以通过单模态特征的加权平均获得。第二种基线方法通过将言论因式分解成单模态、双模态和三模态因子来捕获更丰富的表示。实验显示出在两个任务上的强大性能以及在推理期间相对于神经模型的显着（10倍）加速。

多模态话语嵌入的强大而简单的基准线